{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=\"rtl\" align=\"center\">\n",
    "<font face=\"XB Zar\" size=5>\n",
    "    <font face=\"IranNastaliq\" size=5>\n",
    "      به نام خدا\n",
    "    </font>\n",
    "    <br>\n",
    "    <font size=3>\n",
    "      دانشگاه صنعتی شریف - دانشکده مهندسی کامپیوتر\n",
    "    </font>\n",
    "    <br>\n",
    "    <font color=blue size=5>\n",
    "      مقدمه‌ای بر یادگیری ماشین\n",
    "    </font>\n",
    "    <br>\n",
    "    <hr/>\n",
    "    <font color=red size=6>\n",
    "      فصل دوم: رگرسیون خطی\n",
    "    </font>\n",
    "    <br>\n",
    "      نویسندگان:‌ سید امیرمحمد عیسی زاده\n",
    "    <hr>\n",
    "<br>\n",
    "  <div align=\"right\">\n",
    "  <font color=\"red\" size=5>فهرست مطالب</font>\n",
    "\t<ul>\n",
    "        <li>\n",
    "        <a href=\"#lr_from_scratch\">\n",
    "            رگرسیون خطی از پایه\n",
    "        </a>\n",
    "        <ul>\n",
    "            <br>\n",
    "            <li>\n",
    "                <a href=\"#lr_closed_form\">\n",
    "                  فرم بسته رگرسیون خطی\n",
    "                </a>\n",
    "            </li>\n",
    "            <br>\n",
    "            <li>\n",
    "                <a href=\"#lr_optimization\">\n",
    "                  رگرسیون خطی با استفاده از optimization function\n",
    "                </a>\n",
    "            </li>\n",
    "            <br>\n",
    "            <li>\n",
    "                <a href=\"#gradient_descent\">\n",
    "                  Gradient Descent\n",
    "                </a>\n",
    "            </li>\n",
    "            <br>\n",
    "            <li>\n",
    "                <a href=\"#model_training\">\n",
    "                  آموزش مدل\n",
    "                </a>\n",
    "            </li>\n",
    "        </ul>\n",
    "    </li>\n",
    "    <br>\n",
    "    <li>\n",
    "        <a href=\"#lr_sklearn\">\n",
    "            رگرسیون خطی با استفاده از sklearn\n",
    "        </a>\n",
    "    </li>\n",
    "    <br>\n",
    "    <li>\n",
    "      <a href=\"#beyond_linear\">\n",
    "        فراتر از linear regression\n",
    "      </a>\n",
    "        <ul>\n",
    "            <br>\n",
    "            <li>\n",
    "                <a href=\"#lasso\">\n",
    "            Lasso Regression\n",
    "                </a>\n",
    "            </li>\n",
    "            <br>\n",
    "            <li>\n",
    "                <a href=\"#ridge\">\n",
    "            Ridge Regression\n",
    "                </a>\n",
    "            </li>\n",
    "            <br>\n",
    "            <li>\n",
    "                <a href=\"#elasticnet\">\n",
    "                    Elastic Net\n",
    "                </a>\n",
    "            </li>\n",
    "            <br>\n",
    "            <li>\n",
    "                <a href=\"#ridge_vs_lasso\">\n",
    "                    Ridge vs Lasso\n",
    "                </a>\n",
    "            </li>\n",
    "        </ul>\n",
    "\t</li>\n",
    "    <br>\n",
    "    <br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sIGyifOmnOYc"
   },
   "source": [
    "\n",
    "  <div dir=rtl id=\"lr_from_scratch\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "      <font color=\"red\" size=5>\n",
    "        رگرسیون خطی از پایه\n",
    "      </font>\n",
    "      <hr />\n",
    "در این قسمت ما از یک دیتاست ساده برای تخمین میزان محصول دو میوه(سیب  و پرتقال) با توجه به پارامترهای دما، میزان بارش و رطوبت در چندین منطقه استفاده میکنیم.\n",
    "    <br>\n",
    "      به این منظور دیتاست ساده‌ای را میسازیم:\n",
    "    </font>\n",
    "      \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We use numpy to define our inputs and targets\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dataset (temp, rainfall, humidity, apples, oranges)\n",
    "df = np.array([[73, 67, 43, 56, 70], \n",
    "                   [91, 88, 64, 81, 101], \n",
    "                   [87, 134, 58, 119, 133], \n",
    "                   [102, 43, 37, 22, 37], \n",
    "                   [69, 96, 70, 103, 119]], dtype='float32')\n",
    "\n",
    "# reigons which are used as indexes\n",
    "regions = [\"Semnan\", \"Golestan\", \"Gilan\", \"Ghazvin\", \"Mazandaran\"]\n",
    "\n",
    "columns = [\"Temp(F)\", \"Rainfall(mm)\", \"Humidity(%)\", \"Apples(ton)\", \"Oranges(ton)\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "  <div dir=rtl>\n",
    "    <font face=\"XB Zar\" size=4>\n",
    "برای نمایش دیتاست از کتابخانه pandas استفاده میکنیم:\n",
    "    </font>\n",
    "      \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Temp(F)</th>\n",
       "      <th>Rainfall(mm)</th>\n",
       "      <th>Humidity(%)</th>\n",
       "      <th>Apples(ton)</th>\n",
       "      <th>Oranges(ton)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Semnan</th>\n",
       "      <td>73.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>70.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Golestan</th>\n",
       "      <td>91.0</td>\n",
       "      <td>88.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>101.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gilan</th>\n",
       "      <td>87.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>119.0</td>\n",
       "      <td>133.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ghazvin</th>\n",
       "      <td>102.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>37.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mazandaran</th>\n",
       "      <td>69.0</td>\n",
       "      <td>96.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>103.0</td>\n",
       "      <td>119.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Temp(F)  Rainfall(mm)  Humidity(%)  Apples(ton)  Oranges(ton)\n",
       "Semnan         73.0          67.0         43.0         56.0          70.0\n",
       "Golestan       91.0          88.0         64.0         81.0         101.0\n",
       "Gilan          87.0         134.0         58.0        119.0         133.0\n",
       "Ghazvin       102.0          43.0         37.0         22.0          37.0\n",
       "Mazandaran     69.0          96.0         70.0        103.0         119.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating dataframe using variable columns as columns and regions as index\n",
    "df = pd.DataFrame(df, columns=columns, index=regions)\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "در رگرسیون خطی، هر مقدار خروجی(ستون‌های سیب و پرتقال)، به صورت جمع وزن‌داری از ستون‌های ورودی به اضافه یک مقدار ثابت در نظر گرفته میشوند:\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "apple  = w11 * temp + w12 * rainfall + w13 * humidity + b1\n",
    "orange = w21 * temp + w22 * rainfall + w23 * humidity + b2\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "که به‌این معناست که خروجی‌های ما یک تابع خطی یا یک صفحه از ورودی‌ها است:\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"center\">\n",
    "<img src=\"linear-regression-graph.png\" width=\"500\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 143
    },
    "execution": {
     "iopub.execute_input": "2022-09-08T17:23:58.094178Z",
     "iopub.status.busy": "2022-09-08T17:23:58.093871Z",
     "iopub.status.idle": "2022-09-08T17:23:58.223769Z",
     "shell.execute_reply": "2022-09-08T17:23:58.222728Z",
     "shell.execute_reply.started": "2022-09-08T17:23:58.094152Z"
    },
    "id": "FWaKYE6Zego-",
    "outputId": "33aced47-2337-49da-b453-5f17ef3e90d8"
   },
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "عملا فرم ماتریس خروجی(Y) به این شکل است:\n",
    "      <br>\n",
    "      <br>\n",
    "      <font color=\"black\" size=5>\n",
    "          $$X \\qquad * \\qquad W \\qquad + \\qquad b \\\\$$\n",
    "          $$\n",
    "\\begin{bmatrix}\n",
    "      73 & 67 & 43 \\\\\n",
    "      91 & 88 & 64 \\\\\n",
    "      \\vdots & \\vdots & \\vdots \\\\\n",
    "      69 & 96 & 70 \\\\\n",
    "\\end{bmatrix}\n",
    "*\n",
    "\\begin{bmatrix}\n",
    "      w_{11} & w_{21} \\\\\n",
    "      w_{12} & w_{22} \\\\\n",
    "      w_{13} & w_{23} \\\\\n",
    "\\end{bmatrix}\n",
    "+\n",
    "\\begin{bmatrix}\n",
    "      b_1 & b_2 \\\\\n",
    "      b_1 & b_2 \\\\\n",
    "      \\vdots & \\vdots \\\\\n",
    "      b_1 & b_2 \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "          </font>\n",
    "    </font>\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "      دیتاست را به دو بخش X و Y قسمت میکنیم که X همان ورودی‌های ما و Y میزان بهره‌برداری از میوه‌ها یا همان خروجی‌ها هستند.\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.iloc[:, :3].to_numpy()\n",
    "Y = df.iloc[:, 3:].to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "نکته شایان توجه این است که b فقط مقداری ثابت به حاصل ضرب اضافه می‌کند، پس اگر به ابتدای تمامی سطرهای X مقدار 1 را اضافه کنیم و به ابتدای هر ستون ماتریس W مقدار بایاس متناظر با آن را اضافه کنیم خروجی هیچ تغییری نخواهد کرد.\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1.,  73.,  67.,  43.],\n",
       "       [  1.,  91.,  88.,  64.],\n",
       "       [  1.,  87., 134.,  58.],\n",
       "       [  1., 102.,  43.,  37.],\n",
       "       [  1.,  69.,  96.,  70.]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = np.append(np.ones((5, 1)), X , axis=1)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "بدست آوردن بهترین وزن در رگرسیون خطی، از دو راه ممکن است:\n",
    "      <ol>\n",
    "            <li>فرم بسته: اگر مشتق جزئی بر حسب وزن را مساوی صفر قرار دهیم، با بدست آوردن ماتریس وزن متناظر، به بهترین پاسخ می‌رسیم.</li>\n",
    "            <li>استفاده از optimization function: همچنین می‌توانیم بجای استفاده از فرم بسته، مانند اکثر مدل‌ها، مرحله مرحله جلو رفته و به بهترین وزن، نزدیک شویم.\n",
    "          دقت کنید که بر خلاف رگرسیون خطی، در اکثر مدل‌ها نمی‌توان فرم بسته‌ای از پاسخ پیدا کرد.</li>\n",
    "        </ol>\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "  <div dir=rtl id=\"lr_closed_form\" align=\"justify\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "      <font color=\"red\" size=5>\n",
    "        فرم بسته رگرسیون خطی\n",
    "      </font>\n",
    "      <hr />\n",
    "      همانطور که در کلاس درس گفته شد، پاسخ فرم بسته بدین شکل است:\n",
    "      <br>\n",
    "      </font>\n",
    "  </div>\n",
    "      <div class=\"cmath\"> W = (X<sup>T</sup>X)<sup>-1</sup>X<sup>T</sup>Y</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normal_equation(X, Y):\n",
    "    W = np.dot((np.linalg.inv(np.dot(X.T,X))), np.dot(X.T,Y))\n",
    "    return W"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "این تابع همان معادله بالاست که با گرفتن ورودی X و Y،خروجی W را به ما می‌دهد.\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# .dot() represent matrix multiplication in numpy\n",
    "def lr_predict(X, W):\n",
    "    return np.dot(X, W)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "این تابع برای محاسبه مقادیر Y با استفاده از X و وزن‌های فعلی است.\n",
    "      <br>\n",
    "      حال دو تابع بالا را در یک مدل تعریف می‌کنیم. ابتدا بهترین وزن را بدست می‌آوریم و سپس مقدار Y را پیش‌بینی می‌کنیم.\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def closed_form_lr(X, Y):\n",
    "    W = normal_equation(X, Y)\n",
    "    predictions = lr_predict(X, W)\n",
    "    return predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = closed_form_lr(X, Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2022-09-08T17:23:58.275887Z",
     "iopub.status.busy": "2022-09-08T17:23:58.275573Z",
     "iopub.status.idle": "2022-09-08T17:23:58.309198Z",
     "shell.execute_reply": "2022-09-08T17:23:58.308143Z",
     "shell.execute_reply.started": "2022-09-08T17:23:58.275853Z"
    },
    "id": "-rUt-7j7egpD",
    "outputId": "a4f8f030-0e66-4a95-92a5-79d01e0460d5"
   },
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "حال معیاری برای میزان خطای مدل، تعریف میکنیم.\n",
    "      <br>\n",
    "      ما از معیار MSE(mean squared error) استفاده میکنیم که اخلاف توان دو پیش بینی‌ها نسبت به مقدار واقعی خروجی‌ها است.\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MSE loss\n",
    "def mse_loss(pred, real):\n",
    "    diff = pred - real\n",
    "    return np.sum(diff * diff) / diff.size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "تابع sum تمامی عناصر آن آرایه را با هم جمع میکند.\n",
    "      در تابع mse_loss در ابتدا اختلاف‌ها را بین دو ماتریس پیدا میکنیم، سپس تمامی این اختلاف‌ها را با هم جمع میکنیم و در انتها تقسیم بر تعداد عناصر میکنیم.\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4831212220582056"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mse_loss(predictions, Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BmZv9X3O0cxg"
   },
   "source": [
    "\n",
    "  <div dir=rtl id=\"lr_optimization\" align=\"justify\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "      <font color=\"red\" size=5>\n",
    "        رگرسیون خطی با استفاده از optimization function\n",
    "      </font>\n",
    "      <hr />\n",
    "      آموزش مدل رگرسیون خطی، عملا همان پیدا کردن مقدار بهینه برای وزن‌ها(w11 و w12 و ... ) است. وزن‌های آموزش دیده شده، برای پیشبینی داده‌های جدید استفاده خواهند شد. سپس این کار را تکرار می‌کنیم تا رفته رفته نتایج ما بهتر شوند. برای بهتر کردن مدل، که همان پیدا کردن وزن‌های بهتر باشد، از optimizer استفاده میکنیم. optimizer مورد استفاده در این تمرین، gradient descent است.\n",
    "  </font>\n",
    "  </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "حال باید به وزن‌ها مقدار اولیه بدهیم. روش‌های مختلفی برای این کار وجود دارند که ما از مقداردهی رندوم استفاده میکنیم:\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.3340964 , 0.15729811],\n",
       "       [0.43118427, 0.27434878],\n",
       "       [0.68027836, 0.50680122],\n",
       "       [0.54042591, 0.32601294]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Weights\n",
    "W = np.random.rand(4, 2)\n",
    "W"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "از تابع lr_predict که بالا تعریف کردیم، استفاده می‌کنیم.\n",
    "      <br>\n",
    "      این مدل را روی داده‌های ورودی، که همان X و W هستند تست میکنیم:\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[100.62751237,  68.1589975 ],\n",
       "       [134.02361888,  90.58637301],\n",
       "       [160.34913071, 110.84575633],\n",
       "       [ 93.56262032,  61.99580526],\n",
       "       [133.22234715,  90.56118721]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = lr_predict(X, W)\n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "از تابع mse_loss که بالا تعریف کردیم، استفاده می‌کنیم.\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1458.3668306724899\n"
     ]
    }
   ],
   "source": [
    "loss = mse_loss(predictions, Y)\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "همانطور که از خود مقادیر عناصر پیش‌بینی شده واضح بود، اختلاف خیلی زیاد است. این اختلاف به این دلیل است که هیچ بهینه سازی‌ای در وزن‌ها انجام ندادیم و صرفا یک پیش‌بینی رندوم کردیم.\n",
    "در بخش Gradient Descent نحوه بروزرسانی وزن‌ها توضیح داده شده است.\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7u-9szEQ1UST"
   },
   "source": [
    "\n",
    "  <div dir=rtl id=\"gradient_descent\" align=\"justify\">\n",
    "    <font face=\"XB Zar\" size=4>\n",
    "      <font color=\"red\" size=5>Gradient Descent</font>\n",
    "      <hr />\n",
    "      مقدار loss ما یک تابع درجه دو از وزن‌ها است. و وظیفه ما این است که آن را به کمترین مقدار خودش برسانیم.\n",
    "        اگر به شکل پایین دقت کنیم، و با توجه به حساب دیفرانسیل، یک نقطه را در نظر بگیرید. دو حالت برای این نقطه ممکن است:\n",
    "        <br>\n",
    "         شیب یا همان گرادیان مثبت باشد:\n",
    "        <br>\n",
    "        1) اگر مقدار کمی عقب برویم، مقدار تابع کمتر میشود.\n",
    "        <br>\n",
    "        2)  اگر مقدار کمی جلو برویم، مقدار تابع بیشتر میشود.\n",
    "        <br>\n",
    "  </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"center\">\n",
    "<img src=\"positive gradient.png\" width=\"500\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "         گرادیان منفی باشد:\n",
    "      <br>\n",
    "      1) اگر مقدار کمی عقب برویم، مقدار تابع بیشتر میشود.\n",
    "      <br>\n",
    "      2) اگر مقدار کمی جلو برویم، مقدار تابع کمتر میشود.\n",
    "      <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"center\">\n",
    "<img src=\"negative gradient.png\" width=\"500\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "        بنابراین، در هر مرحله مقدار گرادیان را در عدد کوچکی ضرب میکنیم و وزن‌ها را از این مقدار کم میکنیم.\n",
    "      این مقدار کوچک همان learning rate است که یکی از مهم ترین hyperparameter های هر مدل ماشین لرنینگ است.\n",
    "      دو حالت نامطلوب برای learning rate محتمل است:\n",
    "      <br>\n",
    "      <ul>\n",
    "  <li>مقدار کوچک‌تر از حد مطلوب: در این صورت مدل دیر همگرا میشود و مراحل بیشتری نیاز است، چون در هر مرحله مقدار بسیار کمی تغییر میکند.</li>\n",
    "  <li>مقدار بیشتر از حد مطلوب: در این صورت مدل در نهایت با حالت مینیمم فاصله زیادی خواهد داشت.</li>\n",
    "</ul>\n",
    "      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"center\">\n",
    "<img src=\"gradient_descent.avif\" width=\"500\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "      حال که مقدار خطا را بدست آوردیم و وزن‌های فعلی را داریم، باید مقدار گرادیان را حساب کنیم.\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_gradient(X, error):\n",
    "    gradient = np.dot(X.T, error)\n",
    "    return gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "رابطه نوشته شده همان مقدار گرادیان است. پیشنهاد می‌شود که صحت این رابطه را بررسی کنید.\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "بعد از بدست آوردن گرادیان، باید وزن‌ها را بروزرسانی کنیم.\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_weights(W, lr, gradient):\n",
    "    new_weights = W - lr * gradient\n",
    "    return new_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1ElZ0H5Fg2yw"
   },
   "source": [
    "\n",
    "  <div dir=\"rtl\" id=\"model_training\">\n",
    "    <font face=\"XB Zar\" size=4>\n",
    "      <font color=\"red\" size=5>\n",
    "        آموزش مدل\n",
    "      </font>\n",
    "      <hr />\n",
    "       حال که همه بخش‌های مدل را بررسی کردیم، نوبت آزمایش آن است.\n",
    "        یک بار تمامی مراحل را مرور میکنیم:\n",
    "        <ol>\n",
    "            <li>تولید پیش بینی با استفاده از وزن‌های فعلی</li>\n",
    "            <li>محاسبه مقدار loss</li>\n",
    "            <li>محاسبه گرادیان‌ها با توجه به وزن‌ها</li>\n",
    "            <li>آپدیت کردن وزن‌ها</li>\n",
    "        </ol>\n",
    "        <br>\n",
    "        این کار‌ها را در چندین مرحله(epoch) انجام میدهیم.\n",
    "        <br>\n",
    "        حال باید تابعی را تعریف کنیم که این مراحل را انجام دهد.\n",
    "       </font>\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model_lr_gd(X, Y, W, n_epoches, lr):\n",
    "    losses = []\n",
    "    for i in range(n_epoches):\n",
    "        # generating the predictions \n",
    "        predictions = lr_predict(X, W)\n",
    "        error = predictions - Y\n",
    "        # calculating the loss\n",
    "        loss = mse_loss(predictions, Y)\n",
    "        # adding the loss to our loss list \n",
    "        losses.append(loss)\n",
    "        # calculating gradients\n",
    "        gradient = calc_gradient(X, error) / Y.size\n",
    "        # updating weights and biases\n",
    "        W = update_weights(W, lr, gradient)\n",
    "    return W, losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "         مدلمان را در 10000 مرحله تست میکنیم و مقدار learning rate را  1e-5 میگذاریم.\n",
    "      <br>\n",
    "      همچنین loss ها را در هر مرحله ذخیره میکنیم.\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5235522542792108\n"
     ]
    }
   ],
   "source": [
    "lr = 1e-5\n",
    "n_epoches = 10000\n",
    "W, losses = train_model_lr_gd(X, Y, W, n_epoches, lr)\n",
    "print(losses[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "         همانطور که مشاهده می‌کنید مقدار loss نهایی خیلی نزدیک به مقدار بدست آمده در حالت بسته است.\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAeJklEQVR4nO3de5RdZZnn8e/vVOUC4ZKEKiEmgQSJKLpUoAbjQh1GkNu4DLOaVlitRGUmY4NX7LFhZA1tu7qXji5paW00Ag26aJRGlKgoRC5D90wTqABCCLcyQFJpQopb5JpQVc/8sd9K9rmkTiWpXbtO1e+z1lm197vfs8+zz07qqfd99363IgIzM7PhVMoOwMzMxj8nCzMza8rJwszMmnKyMDOzppwszMysqfayAyhCR0dHLFiwoOwwzMxayurVq5+JiM5G2yZksliwYAHd3d1lh2Fm1lIkPbmzbe6GMjOzppwszMysKScLMzNrysnCzMyacrIwM7OmnCzMzKwpJwszM2vKySLnlW39fPvmR7h3/fNlh2JmNq44WeS8um2AS27t4YGNW8oOxcxsXHGyaMDPgzIzq+ZkkSOp7BDMzMYlJ4sG/KhZM7NqThY5bleYmTVWWLKQdIWkzZLWNNj2JUkhqSOtS9Ilknok3S/pqFzdpZIeS6+lRcWb53aFmVm1IlsWVwIn1xZKmg+cCKzPFZ8CLEqvZcClqe5s4CLg3cAxwEWSZhUVsIcszMwaKyxZRMQdwHMNNl0MfJnqP+CXAD+KzJ3ATElzgJOAlRHxXEQ8D6ykQQIabR6yMDOrNqZjFpKWABsj4vc1m+YCG3LrvalsZ+WN9r1MUrek7r6+vt2Lz6MWZmYNjVmykLQ38D+B/1XE/iNieUR0RURXZ2fDpwKOfF+jFJOZ2UQxli2LNwELgd9LegKYB9wj6SBgIzA/V3deKttZeTHcsDAza2jMkkVEPBARb4iIBRGxgKxL6aiI2ASsAM5KV0UtBrZExFPATcCJkmalge0TU1nRsRb9EWZmLaXIS2evAf4NOFxSr6Szh6l+I7AO6AF+CJwDEBHPAV8D7k6vv05lBcVc1J7NzFpbe1E7jogzm2xfkFsO4Nyd1LsCuGJUgzMzs13iO7hz3LAwM2vMyaIBD1mYmVVzssjxrLNmZo05WTQQvtPCzKyKk0WO2xVmZo05WTTgMQszs2pOFjlDQxbOFWZm1ZwscjyRoJlZY04WDbgbysysmpNFjq+cNTNrzMmiAV86a2ZWzcnCzMyacrJowGMWZmbVnCxyPGZhZtaYk4WZmTXlZJHj+yzMzBpzsmjAj1U1M6vmZJHjMQszs8aKfAb3FZI2S1qTK/umpIcl3S/p55Jm5rZdIKlH0iOSTsqVn5zKeiSdX1S8eW5YmJlVK7JlcSVwck3ZSuDtEfEO4FHgAgBJRwBnAG9L7/kHSW2S2oDvAacARwBnprqFcMPCzKyxwpJFRNwBPFdTdnNE9KfVO4F5aXkJ8JOI2BoRjwM9wDHp1RMR6yJiG/CTVLdQbliYmVUrc8ziU8Bv0vJcYENuW28q21l5HUnLJHVL6u7r69utgPxYVTOzxkpJFpK+AvQDV4/WPiNieUR0RURXZ2fnHu5rlIIyM5sg2sf6AyV9AvgQcHzsuEZ1IzA/V21eKmOY8tGPragdm5m1uDFtWUg6Gfgy8OGIeCW3aQVwhqRpkhYCi4C7gLuBRZIWSppKNgi+oug4PeusmVm1wloWkq4BjgM6JPUCF5Fd/TQNWJnGB+6MiE9HxIOSrgXWknVPnRsRA2k/nwFuAtqAKyLiweJiLmrPZmatrbBkERFnNii+fJj6fwP8TYPyG4EbRzG0pjxmYWZWzXdw5/hqKDOzxpwsGnDDwsysmpOFmZk15WTRiActzMyqOFnU8LCFmVk9J4sG3K4wM6vmZFHDDQszs3pOFg14yMLMrJqTRQ3fa2FmVs/JogHPDWVmVs3JoobbFWZm9ZwsGvCYhZlZNSeLGh6yMDOr52TRgBsWZmbVnCxqyKMWZmZ1nCwa8JiFmVk1J4tabliYmdVxsmjA91mYmVUrLFlIukLSZklrcmWzJa2U9Fj6OSuVS9Ilknok3S/pqNx7lqb6j0laWlS82z+v6A8wM2tBRbYsrgROrik7H7glIhYBt6R1gFOARem1DLgUsuQCXAS8GzgGuGgowRTKDQszsyqFJYuIuAN4rqZ4CXBVWr4KOC1X/qPI3AnMlDQHOAlYGRHPRcTzwErqE9CokpwrzMxqjfWYxYER8VRa3gQcmJbnAhty9XpT2c7K60haJqlbUndfX9/oRm1mNsmVNsAdEcEo/hEfEcsjoisiujo7O3d7P0KEr501M6sy1sni6dS9RPq5OZVvBObn6s1LZTsrL4yn+zAzqzfWyWIFMHRF01Lghlz5WemqqMXAltRddRNwoqRZaWD7xFRWKDcszMyqtRe1Y0nXAMcBHZJ6ya5q+jpwraSzgSeBj6TqNwKnAj3AK8AnASLiOUlfA+5O9f46ImoHzUc37iJ3bmbWogpLFhFx5k42Hd+gbgDn7mQ/VwBXjGJoTblhYWZWzXdw1/BjVc3M6jlZNOAxCzOzak4WNdyuMDOr52TRgCcSNDOr5mRRy00LM7M6ThYNeMzCzKyak0UNNyzMzOo5WZiZWVNOFjV8n4WZWT0niwY866yZWTUnixpuWJiZ1XOyaMDtCjOzak4WNdywMDOr52TRgIcszMyqOVnU8NVQZmb1nCwa8NxQZmbVnCxquF1hZlbPyaIBj1mYmVUrJVlI+qKkByWtkXSNpOmSFkpaJalH0k8lTU11p6X1nrR9QbGxFbl3M7PWNObJQtJc4HNAV0S8HWgDzgC+AVwcEYcBzwNnp7ecDTyfyi9O9QrlhoWZWbWyuqHagb0ktQN7A08BHwCuS9uvAk5Ly0vSOmn78Sr0kiU3LczMao15soiIjcC3gPVkSWILsBp4ISL6U7VeYG5angtsSO/tT/UPKDbGIvduZtZ6yuiGmkXWWlgIvBGYAZw8CvtdJqlbUndfX98e7GdPIzEzm3jK6IY6AXg8Ivoi4nXgeuBYYGbqlgKYB2xMyxuB+QBp+/7As7U7jYjlEdEVEV2dnZ17GKKbFmZmeWUki/XAYkl7p7GH44G1wG3A6anOUuCGtLwirZO23xoFziHuhoWZWb0RJQtJn5e0nzKXS7pH0om784ERsYpsoPoe4IEUw3LgL4HzJPWQjUlcnt5yOXBAKj8POH93PnfXYiz6E8zMWkt78yoAfCoiviPpJGAW8HHgx8DNu/OhEXERcFFN8TrgmAZ1XwP+dHc+Z3d4zMLMrN5Iu6GGfoWeCvw4Ih5kAvfYuGVhZlZtpMlitaSbyZLFTZL2BQaLC6s8mrg50Mxst420G+ps4F3Auoh4RdJs4JOFRVUyzzprZlZtpC2L9wCPRMQLkj4GXEh2c9yE4zELM7N6I00WlwKvSHon8CXgD8CPCouqZB6zMDOrNtJk0Z/ubVgCfDcivgfsW1xY5XHDwsys3kjHLF6UdAHZJbPvk1QBphQXVrncsDAzqzbSlsVHga1k91tsIpuO45uFRVUiP4PbzKzeiJJFShBXA/tL+hDwWkR4zMLMbJIY6XQfHwHuIruT+iPAKkmnD/+u1uVLZ83Mqo10zOIrwH+IiM0AkjqB37HjYUUThnuhzMzqjXTMojKUKJJnd+G9rccNCzOzKiNtWfxW0k3ANWn9o8CNxYRULrcszMzqjShZRMT/kPQnZA8pAlgeET8vLqxyuWFhZlZtpC0LIuJnwM8KjGVc8ESCZmb1hk0Wkl6k8R/aAiIi9iskqpIV+CA+M7OWNGyyiIgJOaXHcDxmYWZWb+Je0bQH3K4wM6tWSrKQNFPSdZIelvSQpPdImi1ppaTH0s9Zqa4kXSKpR9L9ko4qNLYid25m1qLKall8B/htRLwFeCfwEHA+cEtELAJuSesApwCL0msZ2XTphfKQhZlZtTFPFpL2B94PXA4QEdsi4gWy6c+vStWuAk5Ly0uAH0XmTmCmpDkFxlfUrs3MWlYZLYuFQB/wj5LulXSZpBnAgRHxVKqzCTgwLc8FNuTe35vKCuOGhZlZtTKSRTtwFHBpRBwJvMyOLicguyaXXfydLWmZpG5J3X19fbsdnNsVZmb1ykgWvUBvRKxK69eRJY+nh7qX0s+huag2AvNz75+XyqpExPKI6IqIrs7Ozj0K0PdZmJlVG/NkkZ6NsUHS4anoeGAtsAJYmsqWAjek5RXAWemqqMXAllx31ehz08LMrM6Ip/sYZZ8FrpY0FVgHfJIscV0r6WzgSbLnZkA2YeGpQA/wSqpbKLcrzMyqlZIsIuI+oKvBpuMb1A3g3KJjGlKR3A1lZlbDd3DXaJMYHCw7CjOz8cXJooYEg25ZmJlVcbKoUZEYdK4wM6viZFGjUnHLwsyslpNFjaxl4WRhZpbnZFHD3VBmZvWcLGpU5Du4zcxqOVnUcDeUmVk9J4saFYkB90OZmVVxsqiRXQ1VdhRmZuOLk0UNT/dhZlbPyaKGr4YyM6vnZFHD032YmdVzsqhRkRh008LMrIqTRY22iruhzMxqOVnUqLgbysysjpNFDXmA28ysjpNFjYrwmIWZWQ0nixrZmIWThZlZXmnJQlKbpHsl/SqtL5S0SlKPpJ9KmprKp6X1nrR9QcFxOVmYmdUos2XxeeCh3Po3gIsj4jDgeeDsVH428HwqvzjVK0x2B3eRn2Bm1npKSRaS5gH/GbgsrQv4AHBdqnIVcFpaXpLWSduPT/UL4auhzMzqldWy+Dvgy8BgWj8AeCEi+tN6LzA3Lc8FNgCk7VtS/SqSlknqltTd19e324FVJAacLMzMqox5spD0IWBzRKwezf1GxPKI6IqIrs7Ozt3eT3YH9ygGZmY2AbSX8JnHAh+WdCowHdgP+A4wU1J7aj3MAzam+huB+UCvpHZgf+DZooLzk/LMzOqNecsiIi6IiHkRsQA4A7g1Iv4MuA04PVVbCtyQllekddL2W6PA3+aeddbMrN54us/iL4HzJPWQjUlcnsovBw5I5ecB5xcZRKWCxyzMzGqU0Q21XUTcDtyeltcBxzSo8xrwp2MVU3ulQv+ABy3MzPLGU8tiXJg+pcJrrztZmJnlOVnUmNbextb+AQ9ym5nlOFnUmD6lwmBAv0e5zcy2c7KoMa29DYDXXh8oORIzs/HDyaLGtCnZV7K13+MWZmZDnCxqTHfLwsysjpNFDbcszMzqOVnU2GtK1rJ4eWt/k5pmZpOHk0WNjn2nAfDsS9tKjsTMbPxwsqjRuU+WLPpe2lpyJGZm44eTRY2OoWTxopOFmdkQJ4sae01tY7/p7Wza8lrZoZiZjRtOFg0s6JjBE8++XHYYZmbjhpNFAwsOmMHjzzhZmJkNcbJoYGHHDDa+8Cpb+31jnpkZOFk0tLBjBhGw/tlXyg7FzGxccLJoYGHHDAD+0OeuKDMzcLJoaNGB+yDBI5teLDsUM7NxYcyThaT5km6TtFbSg5I+n8pnS1op6bH0c1Yql6RLJPVIul/SUUXHuPfUdhYcMIOHnvpj0R9lZtYSymhZ9ANfiogjgMXAuZKOAM4HbomIRcAtaR3gFGBRei0DLh2LIN9y0L48vMnJwswMSkgWEfFURNyTll8EHgLmAkuAq1K1q4DT0vIS4EeRuROYKWlO0XG+dc5+PPncK55Q0MyMkscsJC0AjgRWAQdGxFNp0ybgwLQ8F9iQe1tvKqvd1zJJ3ZK6+/r69ji2txy0LxHw6NMetzAzKy1ZSNoH+BnwhYio6u+JiAB26SHYEbE8Iroioquzs3OP43vrnP0AWPPv7ooyMyslWUiaQpYoro6I61Px00PdS+nn5lS+EZife/u8VFaoebP2onPfadz75PNFf5SZ2bhXxtVQAi4HHoqIb+c2rQCWpuWlwA258rPSVVGLgS257qoi4+Tog2exer2ThZlZGS2LY4GPAx+QdF96nQp8HfigpMeAE9I6wI3AOqAH+CFwzlgFetQhM3ny2Vc8XbmZTXrtY/2BEfGvgHay+fgG9QM4t9CgduLoQ2YBcM/65znpbQeVEYKZ2bjgO7iH8bY37s+09gr/9odnyw7FzKxUThbDmD6ljcWHHsAdj+75pbhmZq3MyaKJ4w7vZN0zL3sGWjOb1Jwsmjju8DcAcPujm5vUNDObuJwsmlhwwN4c2jGD3zywqexQzMxK42TRhCQ+/K43cufjz7Jpy2tlh2NmVgonixFY8q65RMCK3xd+47iZ2bjkZDECCztmcNTBM/mnVesZHNylKavMzCYEJ4sR+uSxC3ni2Ve45WEPdJvZ5ONkMUKnvP0g5s7ci0tv7yG7qdzMbPJwshih9rYK5/ynN3HP+hf47RpfGWVmk4uTxS74aNd8Dj9wX/72Nw/5CXpmNqk4WeyC9rYKXzvt7fQ+/ypf/eWDZYdjZjZmnCx20TELZ3POcW/i2u5ervy/j5cdjpnZmBjzKcongi+e8GYee/olvvqrtbS3VfjY4kPKDsnMrFBuWeyG9rYKl5x5JMe9uZMLf7GGC3/xgMcwzGxCc7LYTdOntPHDs7r4b+9byNWr1nPixXdwzV3r2do/UHZoZmajThPxnoGurq7o7u4es89b/eRzfPWXa7m/dwuzZ0zllLcfxAlHHMjRh8xiv+lTxiwOM7M9IWl1RHQ13NYqyULSycB3gDbgsoj4+s7qjnWyAIgI/uWxZ/jn1b38bu3TvPr6ABXBYW/Yh8PesA+HduzDwbP3pnPfaXTsM43Ofacxc+8pTGuvIO3sKbNmZmNnuGTREgPcktqA7wEfBHqBuyWtiIi15Ua2gyTe/+ZO3v/mTl7dNsA965/nrsef44GNW1j773/kt2s20WhaqbaKmDG1jX2mtTMjvaZPqTC1vY2pbRWmtVeY2l5halv6WbPeXhFtFWU/2yq0STvK2kSlZr2tktXZsZ5tr6SyikRF2fFIbF+vpHXVrFckxFA9oUp1mWrqDu3bzFpLSyQL4BigJyLWAUj6CbAEGDfJIm+vqW0ce1gHxx7Wsb1sa/8Am/+4lb6XtvLMi9nPLa++zstb+3l56wAvbe3n5a39vLS1n9deH2DLK9vYNhBs6x9g28Ag2/pzr4FBXh9ojRbhzlRyiUcIUv4QoO3L2YJy27J1Va1Xv3dk71HNm6X6bc0+n1z9nb2nCEWm2iITeaF/IrTg913Ud/3WOfvx92ceOer7bZVkMRfYkFvvBd6dryBpGbAM4OCDDx67yEZoWnsb82fvzfzZe4/K/gYHg20DgwxG0D8YDAykn4PBQAytDzIwmCuvWc7Wd9QZHAwCGIxgMLKutYgd64MRubId60N1d5Sln+TWG+x7MLcvgCBg+3L6mbpJo648957cetV7mtSt3l6zbUTvGT7mIhT5J0KRPdLFxt2C33eBX8j8WXsVst9WSRZNRcRyYDlkYxYlh1O4SkVMr7SVHYaZTRKtcunsRmB+bn1eKjMzszHQKsnibmCRpIWSpgJnACtKjsnMbNJoiW6oiOiX9BngJrJLZ6+ICM/kZ2Y2RloiWQBExI3AjWXHYWY2GbVKN5SZmZXIycLMzJpysjAzs6acLMzMrKmWmUhwV0jqA57cg110AM+MUjitYrId82Q7XvAxTxZ7csyHRERnow0TMlnsKUndO5t5caKabMc82Y4XfMyTRVHH7G4oMzNrysnCzMyacrJobHnZAZRgsh3zZDte8DFPFoUcs8cszMysKbcszMysKScLMzNryskiR9LJkh6R1CPp/LLj2ROS5ku6TdJaSQ9K+nwqny1ppaTH0s9ZqVySLknHfr+ko3L7WprqPyZpaVnHNBKS2iTdK+lXaX2hpFXpuH6aprhH0rS03pO2L8jt44JU/oikk0o6lBGRNFPSdZIelvSQpPdMgnP8xfRveo2kayRNn2jnWdIVkjZLWpMrG7XzKuloSQ+k91yikTzjNbY/KnNyv8imPv8DcCgwFfg9cETZce3B8cwBjkrL+wKPAkcA/xs4P5WfD3wjLZ8K/IbskcOLgVWpfDawLv2clZZnlX18wxz3ecA/Ab9K69cCZ6Tl7wN/npbPAb6fls8AfpqWj0jnfhqwMP2baCv7uIY53quA/5qWpwIzJ/I5JnvE8uPAXrnz+4mJdp6B9wNHAWtyZaN2XoG7Ul2l957SNKayv5Tx8gLeA9yUW78AuKDsuEbx+G4APgg8AsxJZXOAR9LyD4Azc/UfSdvPBH6QK6+qN55eZE9QvAX4APCr9B/hGaC99hyTPRvlPWm5PdVT7XnP1xtvL2D/9ItTNeUT+RzPBTakX4Dt6TyfNBHPM7CgJlmMynlN2x7OlVfV29nL3VA7DP0jHNKbylpeanofCawCDoyIp9KmTcCBaXlnx99K38vfAV8GBtP6AcALEdGf1vOxbz+utH1Lqt9Kx7sQ6AP+MXW9XSZpBhP4HEfERuBbwHrgKbLztpqJfZ6HjNZ5nZuWa8uH5WQxwUnaB/gZ8IWI+GN+W2R/VkyIa6clfQjYHBGry45lDLWTdVVcGhFHAi+TdU9sN5HOMUDqp19ClijfCMwATi41qBKUcV6dLHbYCMzPrc9LZS1L0hSyRHF1RFyfip+WNCdtnwNsTuU7O/5W+V6OBT4s6QngJ2RdUd8BZkoaeiJkPvbtx5W27w88S+scL2R/EfZGxKq0fh1Z8pio5xjgBODxiOiLiNeB68nO/UQ+z0NG67xuTMu15cNystjhbmBRuqpiKtlg2IqSY9pt6eqGy4GHIuLbuU0rgKGrIpaSjWUMlZ+VrqxYDGxJTd6bgBMlzUp/1Z2YysaViLggIuZFxAKyc3drRPwZcBtweqpWe7xD38PpqX6k8jPSVTQLgUVkg4HjTkRsAjZIOjwVHQ+sZYKe42Q9sFjS3unf+NAxT9jznDMq5zVt+6Okxek7PCu3r50rexBnPL3Irip4lOzKiK+UHc8eHst7yZqp9wP3pdepZP21twCPAb8DZqf6Ar6Xjv0BoCu3r08BPen1ybKPbQTHfhw7roY6lOyXQA/wz8C0VD49rfek7Yfm3v+V9D08wgiuEin5WN8FdKfz/Auyq14m9DkGvgo8DKwBfkx2RdOEOs/ANWRjMq+TtSDPHs3zCnSl7+8PwHepuUii0cvTfZiZWVPuhjIzs6acLMzMrCknCzMza8rJwszMmnKyMDOzppwszHIk3S5p1B923+BzPpdmib266M+q+dy/kvQXY/mZNjG0N69iZiMhqT12zE/UzDnACRHR27Sm2TjgloW1HEkL0l/lP0zPNbhZ0l5p2/aWgaSONP0Hkj4h6RfpOQBPSPqMpPPSBHx3Spqd+4iPS7pP2fMSjknvn5GeMXBXes+S3H5XSLqV7Iap2ljPS/tZI+kLqez7ZDeR/UbSF2vqt0n6pqS707MJ/nsqP07SHZJ+rez5C9+XVEnbzkzPJlgj6Ru5fZ0s6R5Jv5eUj+2I9D2tk/S5XP2PpeO7T9IPUixtkq5M+36gNl6bRMq+U9Evv3b1RTZ1cz/wrrR+LfCxtHw76Q5WoAN4Ii1/guwu1n2BTrLZRz+dtl1MNtHi0Pt/mJbfT5oiGvjb3GfMJLvTf0baby/pbtqaOI8mu6N2BrAP8CBwZNr2BNDR4D3LgAvT8jSyu7MXkt2V/hpZkmkDVpJNX/FGsikwOsl6Cm4FTkvrG4CFaV9Dd/v+FfD/0r47yOZJmgK8FfglMCXV+weyaSCOBlbm4ptZ9vn3q5yXu6GsVT0eEfel5dVkCaSZ2yLiReBFSVvIfjlC9gv9Hbl61wBExB2S9pM0k2xenQ/n+vunAwen5ZUR8VyDz3sv8POIeBlA0vXA+4B7h4nxROAdkobmOdqfbN6ibcBdEbEu7euatP/Xgdsjoi+VX02W5AaAOyLi8XQs+fh+HRFbga2SNpNNdX08WWK4O5suiL3IJqr7JXCopL8Hfg3cPEzsNoE5WVir2ppbHiD75QZZi2Ooe3X6MO8ZzK0PUv1/oXYOnCCbf+dPIuKR/AZJ7yabGny0CPhsRFRN5CfpuJ3EtTtqv7v29LlXRcQFdQFJ7yR7wNCngY+QzTdkk4zHLGyieYLsL2TYMQvprvoogKT3ks3guYVsBs/Pplk6kXTkCPbzL8BpaYbUGcB/SWXDuQn4c2XTyyPpzem9AMcomxW5kmL8V7LJ8f5jGp9pI3vq2f8B7gTen2ZUpWZMppFbgNMlvWGovqRDJHUAlYj4GXAh2RToNgm5ZWETzbeAayUtI+s22R2vSbqXrC9/6K/or5E9ie/+9Mv6ceBDw+0kIu6RdCU7pr6+LCKG64ICuIysS+2elJj6yMYgIJtG/7vAYWRTcv88IgYlnZ/WRdbFdANA+g6uT/FuJnus7s5iXSvpQuDmVP914FzgVbIn8Q39YVnX8rDJwbPOmrWA1A31FxExbIIyK4q7oczMrCm3LMzMrCm3LMzMrCknCzMza8rJwszMmnKyMDOzppwszMysqf8PKRzDlwpEmhkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "  \n",
    "# x axis values\n",
    "x = [i for i in range(n_epoches)]\n",
    "# corresponding y axis values\n",
    "y = losses\n",
    "  \n",
    "# plotting the points \n",
    "plt.plot(x, y)\n",
    "  \n",
    "# naming the x axis\n",
    "plt.xlabel('number of epoches')\n",
    "# naming the y axis\n",
    "plt.ylabel('loss')\n",
    "  \n",
    "# function to show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "         همانطور که مشاهده میکنید، در ابتدا به مقدار زیادی از خطا کم میشود و رفته رفته اختلاف خطاها در هر دو مرحله متوالی کمتر میشود تا در نهایت در همسایگی کوچکی از مینیمم، تغییر کند. هر چه learning rate را کمتر کنیم طول این همسایگی کمتر میشود. (البته در مراحل بیشتری باید کار را انجام دهیم تا نتیجه مطلوبی کسب کنیم)\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   <div dir=\"rtl\" id=\"lr_sklearn\">\n",
    "    <font face=\"XB Zar\" size=4>\n",
    "      <font color=\"red\" size=5>\n",
    "        رگرسیون خطی با استفاده از sklearn\n",
    "      </font>\n",
    "        <hr />\n",
    "       <br>\n",
    "      حال دیتاست داده شده را با استفاده از مدل آماده linear regression در sklearn یاد میگیریم.\n",
    "        <br>\n",
    "        این کتابخانه تابع آماده برای محاسبه خطا دارد.\n",
    "       </font>\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model\n",
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.iloc[:, :3].to_numpy()\n",
    "Y = df.iloc[:, 3:].to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "         یک مدل رگرسیون خطی از ماژول linear_model این کتابخانه تعریف می‌کنیم.\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create linear regression object\n",
    "regr = linear_model.LinearRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "با تابع fit، مدل را بر روی داده‌ها آموزش می‌دهیم.    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train the model using the training sets\n",
    "regr.fit(X, Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "با تابع predict، از ورودی‌ها مقدار خروجی‌ها را بدست آورده و سپس میزان خطا را محاسبه می‌کنیم.    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.48312175"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = regr.predict(X)\n",
    "predictions\n",
    "mean_squared_error(Y, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "همانطور که مشاهده می‌کنید میزان خطای بدست آمده بسیار نزدیک به میزان خطا در فرم بسته است!!    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   <div dir=\"rtl\" id=\"beyond_linear\" align=\"justify\">\n",
    "    <font face=\"XB Zar\" size=4>\n",
    "      <font color=\"red\" size=5>\n",
    "        فراتر از linear regression\n",
    "      </font>\n",
    "        <hr />\n",
    "      در رگرسیون خطی‌ای که معرفی کردیم، هیچ قانون محدود کننده‌ای روی وزن ها قرار ندادیم. این کار باعث پیچیده تر شدن مدل می‌شود که به overfitting بر روی داده train منجر می‌شود. به گذاشتن محدودیت روی وزن‌ها regularization می‌گوییم. به عبارتی وظیفه ما مینیمم کردن مجموع loss و regularization term است.\n",
    "        <br>\n",
    "        حال مدل‌هایی را معرفی می‌کنیم که بخش regularization را به linear regression اضافه کرده‌اند.\n",
    "       </font>\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   <div dir=\"rtl\" id=\"lasso\" align=\"justify\">\n",
    "    <font face=\"XB Zar\" size=4>\n",
    "      <font color=\"red\" size=5>\n",
    "        Lasso Regression\n",
    "      </font>\n",
    "        <hr />\n",
    "      در این مدل، regularization term به صورت جمع قدر مطلق وزن‌ها تعریف می‌شود.\n",
    "        به عبارتی در این مدل باید تابع زیر را کمینه کنیم:\n",
    "        <br>\n",
    "        <br>\n",
    "        <font color=\"black\" size=5>\n",
    "        $$\\frac{1}{2m} \\Sigma_{i=1}^{m}{(y-Xw)^2} + \\alpha \\Sigma_{j=1}^{p}{\\mid w_j\\mid}$$\n",
    "            </font>\n",
    "       </font>\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "در lasso، به علت پنالتی‌ای که روی وزن ها قرار دارد، تمایل به کم کردن وزن ها به صورت برابر وجود دارد، به همین دلیل ماتریس وزن، sparsity بالایی دارد و تعداد زیادی از وزن ها به صفر میل می‌کنند.    \n",
    "    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   <div dir=\"rtl\" id=\"ridge\" align=\"justify\">\n",
    "    <font face=\"XB Zar\" size=4>\n",
    "      <font color=\"red\" size=5>\n",
    "        Ridge Regression\n",
    "      </font>\n",
    "        <hr />\n",
    "      در این مدل، regularization term به صورت جمع توان دو وزن‌ها تعریف می‌شود.\n",
    "        به عبارتی در این مدل باید تابع زیر را کمینه کنیم:\n",
    "        <br>\n",
    "        <br>\n",
    "        <font color=\"black\" size=5>\n",
    "        $$\\Sigma_{i=1}^{m}{(y-Xw)^2} + \\alpha \\Sigma_{j=1}^{p}{w_j^2}$$\n",
    "            </font>\n",
    "       </font>\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "در ridge، بر خلاف lasso چون پنالتی بر روی توان دو وزن ها قرار دارد، وزن ها صفر نمی‌شوند و با همدیگر کم می‌شوند، چون به عنوان مثال اختلاف توان دو 4 و 2، بیشتر از اختلاف توان دو 2 و صفر هست، به همین علت همه با هم کم می‌شوند.\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   <div dir=\"rtl\" id=\"elasticnet\" align=\"justify\">\n",
    "    <font face=\"XB Zar\" size=4>\n",
    "      <font color=\"red\" size=5>\n",
    "        Elastic Net\n",
    "      </font>\n",
    "        <hr />\n",
    "      در این مدل، regularization term به صورت جمع توان دو و توان یک وزن‌ها تعریف می‌شود. به عبارتی خصوصیت های ridge و lasso را در خود جمع کرده است.\n",
    "        در این مدل باید تابع زیر را کمینه کنیم:\n",
    "        <br>\n",
    "        <br>\n",
    "        <font color=\"black\" size=5>\n",
    "        $$\\frac{1}{2m} \\Sigma_{i=1}^{m}{(y-Xw)^2} + \\alpha * ratio * \\Sigma_{j=1}^{p}{\\mid w_j\\mid} + 0.5 * \\alpha * (1-ratio) * \\Sigma_{j=1}^{p}{w_j^2}$$\n",
    "            </font>\n",
    "       </font>\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "با تابع predict، از ورودی‌ها مقدار خروجی‌ها را بدست آورده و سپس میزان خطا را محاسبه می‌کنیم.\n",
    "    <br>\n",
    "      حال نمونه ای از پیاده‌سازی ElasticNet در پایتون را مشاهده می‌کنیم.\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Regression:\n",
    "    \n",
    "    def __init__(self, regularization, lr, epoch):\n",
    "        self.m = None #samples\n",
    "        self.n = None #features\n",
    "        self.w = None #weight\n",
    "        self.b = None #bias\n",
    "        self.regularization = regularization #penalty object\n",
    "        self.lr = lr #learning rate\n",
    "        self.epoch = epoch #iteration\n",
    "        \n",
    "    def __calculate_cost(self, y, y_pred):\n",
    "        return (1 / (2*self.m)) * np.sum(np.square(y_pred-y)) + self.regularization(self.w)\n",
    "    \n",
    "    def __hypothesis(self, w, X):\n",
    "        return np.dot(X, w) \n",
    "    \n",
    "    def __initialization(self, X):\n",
    "        X = np.insert(X, 0, 1, axis=1)\n",
    "        self.m, self.n = X.shape\n",
    "        self.w = np.zeros((self.n,1))\n",
    "        return X\n",
    "    \n",
    "    def __update_parameters(self, X, y, y_pred):\n",
    "        dw = (1/self.m) * np.dot(X.T, (y_pred - y)) + self.regularization.derivation(self.w)\n",
    "        self.w = self.w - self.lr * dw\n",
    "        return True\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        X = self.__initialization(X)\n",
    "        for e in range(1, self.epoch+1):\n",
    "            y_pred = self.__hypothesis(self.w, X)\n",
    "            cost = self.__calculate_cost(y, y_pred)\n",
    "            self.__update_parameters(X, y, y_pred)\n",
    "            if e % 5000 == 0:\n",
    "#                 print(f\"The Cost in iteration {e}----->{cost} :)\")\n",
    "                pass\n",
    "\n",
    "        return True\n",
    "\n",
    "    def predict(self, X_test):\n",
    "        X_test = np.insert(X_test, 0 , 1, axis= 1)\n",
    "        y_pred = self.__hypothesis(self.w, X_test)\n",
    "        return y_pred\n",
    "\n",
    "class ElasticPenalty:\n",
    "    \n",
    "    def __init__(self, l = 0.1, l_ratio = 0.5):\n",
    "        self.l = l \n",
    "        self.l_ratio = l_ratio\n",
    "\n",
    "    def __call__(self, w):\n",
    "        l1_contribution = self.l_ratio * self.l * np.sum(np.abs(w))\n",
    "        l2_contribution = (1 - self.l_ratio) * self.l * 0.5 * np.sum(np.square(w))\n",
    "        return (l1_contribution + l2_contribution)\n",
    "\n",
    "    def derivation(self, w):\n",
    "        l1_derivation = self.l * self.l_ratio * np.sign(w)\n",
    "        l2_derivation = self.l * (1 - self.l_ratio) * w\n",
    "        return (l1_derivation + l2_derivation)\n",
    "\n",
    "class ElasticNet(Regression):\n",
    "    \n",
    "    def __init__(self, l, l_ratio, lr, epoch):\n",
    "        self.regularization = ElasticPenalty(l,l_ratio)\n",
    "        super().__init__(self.regularization, lr, epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "نکته بسیار مهم این است که اگر مقدار l_ratio در پارامتر ها برابر با صفر باشد، مدل تبدیل به Ridge می‌شود و اگر برابر با یک باشد مدل تبدیل به Lasso می‌شود. (چرا؟)\n",
    "    <br>\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   <div dir=\"rtl\" id=\"ridge_vs_lasso\" align=\"justify\">\n",
    "    <font face=\"XB Zar\" size=4>\n",
    "      <font color=\"red\" size=5>\n",
    "        Ridge vs Lasso\n",
    "      </font>\n",
    "        <hr />\n",
    "            به ازای l_ratio های مختلف(از Ridge تا Lasso) مقدار mse_loss را برای مدلمان بدست می‌آوریم.\n",
    "        <br>\n",
    "        برای ساخت دیتاست از تابع make_regression کتابخانه sklearn استفاده می‌کنیم. در این تابع می‌توانیم میزان نویز را به دلخواه تعیین کنیم.\n",
    "       </font>\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_regression\n",
    "X, y = make_regression(n_samples=50000, n_features=8, noise=5)\n",
    "y = y[:, np.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "losses = []\n",
    "for i in range(11):\n",
    "    parameters = {\n",
    "        \"l_ratio\" : i * 0.1,\n",
    "        \"l\" : 0.1,\n",
    "        \"lr\" : 0.001,\n",
    "        \"epoch\" : 10000\n",
    "    }\n",
    "    model = ElasticNet(**parameters)\n",
    "    model.fit(X, y) \n",
    "    y_pred = model.predict(X)\n",
    "    losses.append(mse_loss(y_pred, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAlsElEQVR4nO3deXhU9f328fcneyAhIRAQCTuobLINEEFc0boVcMFdRFDqVqu1den21Pp7+nSxrbYFlU1wQ0SsYku11qogQiBBAUFAdsKWsGUBEhLyff6YgV+KAQJk5mRm7td15crMOWdm7kOAO2f7HnPOISIiAhDjdQAREak/VAoiInKESkFERI5QKYiIyBEqBREROSLO6wCno2nTpq5t27ZexxARCSt5eXk7nXOZNc0L61Jo27Ytubm5XscQEQkrZrbxWPOCtvvIzCabWYGZfVXDvEfNzJlZ08BzM7M/m9kaM1tqZr2DlUtERI4tmMcUpgBXHD3RzFoBlwObqk2+EugU+BoDPB/EXCIicgxBKwXn3Bxgdw2z/gQ8BlS/lHoo8LLzWwCkm1mLYGUTEZGahfTsIzMbCmxxzi05alZLYHO15/mBaTW9xxgzyzWz3MLCwiAlFRGJTiErBTNrAPwE+MXpvI9zbrxzzuec82Vm1njwXERETlEozz7qALQDlpgZQBaw2Mz6AVuAVtWWzQpMExGREArZloJzbplzrplzrq1zri3+XUS9nXPbgVnAiMBZSNlAkXNuW6iyiYiIXzBPSZ0GzAfONrN8Mxt9nMVnA+uANcAE4P5g5QLYWVrOU+8t52BlVTA/RkQk7ARt95Fz7pYTzG9b7bEDHghWlqPlrNvNS/M2sKv0IM/e1JOYGAvVR4uI1GthfUXzqbr63BZs2n0Ov31/JU1TEvn5NZ0JHOcQEYlqUVkKAPde2J6CkjImz1vPGWmJjLmgg9eRREQ8F7WlYGb8/OouFJSU8+vZK8lMTeTaXllexxIR8VTUlgJATIzxxxt7sLv0ID+esZQmDRO54Cxd+yAi0Svq76eQGBfLiyP60Kl5Kve+msfS/L1eRxIR8UzUlwJAo6R4pt7Vl8YNErjrpUVs2LnP60giIp5QKQQ0a5TEy6P7UeUcIyYvpLCk3OtIIiIhp1KopkNmCpNH9qWwpJy7piyktLzS60giIiGlUjhKr9aNGXdbb77eVsJ9r+bpqmcRiSoqhRpcfE4zfnNdd+Z+s5PH3lpCVZU78YtERCJAVJ+SejzDfa0oKCnn9x+solmjJH5yVWevI4mIBJ1K4Tjuv6gDBcVljJ+zjmapidw9qL3XkUREgkqlcBxmxi++25XC0nL+5x9fk5mayNCeNd4QTkQkIuiYwgnExhh/vLEn/dtl8KMZS5j7jW4BKiKRS6VQC0nxsUy400eHzBTufSWPr7YUeR1JRCQoVAq11Cgpnqmj+pHeIIGRLy1k4y5d9SwikUelcBKaN0pi6qh+VFY57py8kJ2luupZRCKLSuEkdWyWwqQ7+7K9uIxRUxaxT1c9i0gEUSmcgj5tGjP21t4s31rMfa8t1lXPIhIxVAqn6NLOzfn1td2Ys7qQx2cu1VXPIhIRdJ3Cabipb2sKisv5w4eradYokSev1FXPIhLeVAqn6cFLOlJQUs6Ln66jWWoSo89v53UkEZFTplI4TWbGL4d0pbCknKf/voLM1ESG9DjT61giIqdExxTqQGyM8ezNPenXLoNH3/ySz9fs9DqSiMgpUSnUkaT4WCaM8NG+aQpjdNWziIQplUIdSkuOZ8qovjRKimPkS4vYvHu/15FERE5K0ErBzCabWYGZfVVt2u/NbKWZLTWzv5lZerV5T5rZGjNbZWbfCVauYGuRlszUUf2oOFTFiMkL2aWrnkUkjARzS2EKcMVR0z4EujnnzgVWA08CmFkX4Gaga+A148wsNojZgqpT81Qmj/Sxde8BXfUsImElaKXgnJsD7D5q2r+cc4f/h1wAZAUeDwXecM6VO+fWA2uAfsHKFgp92mTw11t7s2xLEQ+8vpiKQ7rqWUTqPy+PKYwC/hl43BLYXG1efmDat5jZGDPLNbPcwsL6fW+Dy7o05/9e251PVhXyxMxlOKernkWkfvOkFMzsp0Al8NrJvtY5N94553PO+TIzM+s+XB27pV9rHhl8FjMX5/P7D1Z5HUdE5LhCfvGamY0ErgEudf/7q/MWoFW1xbIC0yLCQ5d2ZEdJGeM+WUuz1ERGDtRVzyJSP4V0S8HMrgAeA4Y456qfrzkLuNnMEs2sHdAJWBjKbMFkZjw9tBuXd2nOU39fwawlW72OJCJSo2CekjoNmA+cbWb5ZjYa+CuQCnxoZl+a2QsAzrnlwJvACuB94AHn3KFgZfNCbIzx51t60bdNBo9M/5IPlm/3OpKIyLdYOB/89Pl8Ljc31+sYJ6W0vJI7JuXw1ZYiXryjD5ec09zrSCISZcwszznnq2mermgOsZTEOKbc1Y9zzmjEva8s5tPV9fsMKhGJLioFD6Qlx/PK6H50aJbCmJdzNYCeiNQbKgWPpDdI4LW7+9OmSQNGT81l4frdJ36RiEiQqRQ8lNEwgdfuzubM9CTuemkheRv3eB1JRKKcSsFjmamJvH5PNpmpiYycvJAlm/d6HUlEophKoR5o3iiJ1+/JJr1h/JEzk0REvKBSqCfOTE/m9buzSUmM445JOazcXux1JBGJQiqFeqRVRgOmjckmIS6G2yfmsKagxOtIIhJlVAr1TJsmDXn9nmzAuGVCDusKS72OJCJRRKVQD3XITGHaPf2pqnLcOiGHTbt0W08RCQ2VQj3VqXkqr97dn7LKQ9wyYQH5e1QMIhJ8KoV6rHOLRrw6uj8lZRXcOiGHbUUHvI4kIhFOpVDPdWuZxsuj+7N730FunZBDQXGZ15FEJIKpFMJAz1bpTLmrLzuKy7h1Yg47S8u9jiQiEUqlECZ8bTOYPLIv+Xv2c/vEHPbsO+h1JBGJQCqFMJLdvgkTR/Rl3c593D4ph6L9FV5HEpEIo1IIM+d3asr4O/rwzY5SRkzOobhMxSAidUelEIYuOrsZ427rzfKtxYycvJDS8kqvI4lIhFAphKnBXZrzl1t6sSS/iFFTFrH/oIpBRE6fSiGMXdm9BX+6qSe5G3Zz99RcyioOeR1JRMKcSiHMDelxJs8M78H8dbsY80qeikFETotKIQJc1zuL31zXnTmrC3ngtcUcrKzyOpKIhCmVQoS4qW9rnh7WjY9WFvD9aYupOKRiEJGTp1KIIHdkt+EX13Thg+U7eGT6l1SqGETkJMV5HUDq1qjz21FZVcWvZ68kPjaGZ4b3IDbGvI4lImFCpRCBxlzQgYOVVTzzr9XExRi/vf5cYlQMIlILQdt9ZGaTzazAzL6qNi3DzD40s28C3xsHppuZ/dnM1pjZUjPrHaxc0eLBSzrx0KWdmJGXz8/e/QrnnNeRRCQMBPOYwhTgiqOmPQF85JzrBHwUeA5wJdAp8DUGeD6IuaLGI4M7ce+FHXg9ZxNPvbdCxSAiJxS03UfOuTlm1vaoyUOBiwKPpwKfAI8Hpr/s/P9rLTCzdDNr4ZzbFqx80cDMePyKs6k4VMWkz9YTH2v85KrOmGlXkojULNTHFJpX+49+O9A88LglsLnacvmBad8qBTMbg39rgtatWwcvaYQwM352dWcqD1UxYe56DlQc4ldDuukYg4jUyLMDzc45Z2YnvT/DOTceGA/g8/m0P6QWzIxfDulKUnwsL85ZR0lZJc8M70F8rM5IFpH/FupS2HF4t5CZtQAKAtO3AK2qLZcVmCZ1xMx48qrOpDWI53fvr6K0rJKxt/UmKT7W62giUo+E+lfFWcCdgcd3Au9Wmz4icBZSNlCk4wnBcf9FHXl6WDf+s6qAOycvpET3YxCRaoJ5Suo0YD5wtpnlm9lo4DfAZWb2DTA48BxgNrAOWANMAO4PVi7xX/n87E09ydu4h1smLGCX7vksIgEWzqcp+nw+l5ub63WMsPWflTu479XFZDVO5tW7+9MiLdnrSCISAmaW55zz1TRPRxqj2CXnNOflUf3YUVzODc/PZ/3OfV5HEhGPqRSiXP/2TZh2TzYHKg4x/IX5rNha7HUkEfGQSkHonpXGm987j/hY46bx88ndsNvrSCLiEZWCANCxWQoz7j2PpimJ3D4ph09XF3odSUQ8oFKQI7IaN+DN751Hu6Yp3D11EbOX6axgkWijUpD/kpmayBtjsumRlc6Dry9m+qJNXkcSkRBSKci3pCXH8/LofgzqlMnjM5cxfs5aryOJSIioFKRGDRLimDDCx9XdW/Dr2Sv5/QcrNfS2SBTQndfkmBLiYvjzLb1ITYpj7MdrKT5QyVNDumqEVZEIplKQ44qNMf7fdd1JS47nxTnrKC6r0AirIhFMpSAnpBFWRaKHft2TWtMIqyKRT6UgJ6X6CKu3Tshh976DXkcSkTqkUpCTNrRnS8aP6MPqHSUMf+FzthUd8DqSiNQRlYKcEo2wKhKZVApyyjTCqkjkUSnIaak+wurN4+eTt1EjrIqEM5WCnLbDI6w2SUnk9okLNcKqSBhTKUidODzCatumDTXCqkgYUylIndEIqyLhT6UgderoEVYnzFnndSQROQkqBalz1UdY/b+zv+aZD1ZphFWRMFGrUjCzH5hZI/ObZGaLzezyYIeT8HV4hNWb+7birx+v4fGZSzlYWeV1LBE5gdpuKYxyzhUDlwONgTuA3wQtlUSEwyOsPnRJR97Mzef2SRoWQ6S+q20pHB5A/yrgFefc8mrTRI7JzPjh5Wfz3M09+XLzXoaNncc3O0q8jiUix1DbUsgzs3/hL4UPzCwV0L4AqbWhPVsyfUw2+w8e4rpxn/PJqgKvI4lIDWpbCqOBJ4C+zrn9QDxw16l+qJk9YmbLzewrM5tmZklm1s7McsxsjZlNN7OEU31/qZ96tW7Muw8OJCujAaOmLOKleet1AFqknqltKZwHrHLO7TWz24GfAUWn8oFm1hJ4CPA557oBscDNwG+BPznnOgJ78BeRRJiW6cm8de95DO7cnKfeW8FP/vYVFYe00SlSX9S2FJ4H9ptZD+BRYC3w8ml8bhyQbGZxQANgG3AJ8FZg/lRg2Gm8v9RjDRPjeOH2Ptx/UQemLdzEiEkL2btfB6BF6oPalkKl82/nDwX+6pwbC6Seygc657YAzwCb8JdBEZAH7HXOVQYWywdansr7S3iIiTEeu+Ic/nhjD/I27mHY2HmsKSj1OpZI1KttKZSY2ZP4T0X9h5nF4D+ucNLMrDH+cmkHnAk0BK44idePMbNcM8stLNTAa+Huut5ZTBvTn9LySq4dN4+53+hnKuKl2pbCTUA5/usVtgNZwO9P8TMHA+udc4XOuQrgbWAgkB7YnUTg/bfU9GLn3HjnnM8558vMzDzFCFKf9GmTwTsPDKRlejIjX1rEy/M3eB1JJGrVqhQCRfAakGZm1wBlzrlTPaawCcg2swZmZsClwArgY+CGwDJ3Au+e4vtLGMpq3IC37hvAxWdn8ot3l/Pzd3QAWsQLtR3m4kZgITAcuBHIMbMbjv+qmjnncvAfUF4MLAtkGA88DvzQzNYATYBJp/L+Er5SEuN48Q4f37uwPa8s2MhdLy2iaH+F17FEoorV5jxxM1sCXOacKwg8zwT+7ZzrEeR8x+Xz+Vxubq6XESRIZuRu5id/W0arxg2YNLIv7Zo29DqSSMQwszznnK+mebU9phBzuBACdp3Ea0VO2nBfK16/J5u9ByoYNnYen6/Z6XUkkahQ2//Y3zezD8xspJmNBP4BzA5eLBHo2zaDdx8YSPNGidwxeSGv5Wz0OpJIxKvtgeYf49/vf27ga7xz7vFgBhMBaJXRgJn3DeDCszL56d++4pezllOpA9AiQRN34kX8nHMzgZlBzCJSo9SkeCaM8PGbf37NhLnrWVtYyl9v7U1a8ildKiMix3HcLQUzKzGz4hq+SsysOFQhRWJjjJ9e3YXfXt+dBet2cd24eWzYuc/rWCIR57il4JxLdc41quEr1TnXKFQhRQ67qW9rXhndn937DjJs3Dzmr93ldSSRiKIziCTsZLdvwjsPDKRpSiJ3TMrhjYWbvI4kEjFUChKW2jRpyNv3D2Bgx6Y88fYyfvXeCg5V6d4MIqdLpSBhq1FSPJPu9DFqYDsmz1vP6KmLKCnTFdAip0OlIGEtLjaGX3y3C7++tjuffbOT68Z9zqZd+72OJRK2VAoSEW7t35qXR/ejoKScYePmsXD9bq8jiYQllYJEjAEdmvLOAwNJbxDPbRMX8GbuZq8jiYQdlYJElHZNG/K3+weS3b4Jj721lP/5+woNwS1yElQKEnHSkuN5aWRfRg5oy8TP1jP8hfls3q3jDCK1oVKQiBQXG8Mvh3Rl3G29WVtYylXPzeW9JVu9jiVS76kUJKJd1b0Fsx8aRKfmKXx/2hc8MXMpBw4e8jqWSL2lUpCI1yqjAdO/dx4PXNyB6bmb+e5fP2Pldg3dJVITlYJEhfjYGH78nXN4dXR/ig5UMOSv83hlwUZqc+dBkWiiUpCoMrBjU/75g0EM6NCEn7/zFfe+msfe/Qe9jiVSb6gUJOo0TUlk8p19+dnVnfnPygKuem4uizboYjcRUClIlIqJMe4e1J6Z9w0gPi6Gm16cz18++kaD6knUUylIVDs3K52/f/98hvQ4kz98uJrbJi5ge1GZ17FEPKNSkKiXmhTPn27qyTPDe7A0v4grn5vDR1/v8DqWiCdUCiKAmXFDnyze+/75tEhLZvTUXJ56bznllbqmQaKLSkGkmg6ZKbx9/wBGDmjLS/M2cN24z1lXWOp1LJGQUSmIHCUpPpZfDunKhBE+tuw9wDV/+YyZeflexxIJCZWCyDFc1qU5//zBILq1TOPRGUt4ZPqXlJZXeh1LJKg8KQUzSzezt8xspZl9bWbnmVmGmX1oZt8Evjf2IptIdS3Skpl2TzaPDD6Ld7/cwjV/nsuy/CKvY4kEjVdbCs8B7zvnzgF6AF8DTwAfOec6AR8Fnot4LjbG+MHgTky7J5vyyique34eE+euo0rXNEgECnkpmFkacAEwCcA5d9A5txcYCkwNLDYVGBbqbCLH0799E2Y/NIiLzm7G//zja0ZPXcSu0nKvY4nUKS+2FNoBhcBLZvaFmU00s4ZAc+fctsAy24HmNb3YzMaYWa6Z5RYWFoYosohf44YJjL+jD78a2pV5a3dx5XNz+XzNTq9jidQZL0ohDugNPO+c6wXs46hdRc4/dGWN2+bOufHOOZ9zzpeZmRn0sCJHMzNGnNeWd+4fSEpSHLdNyuH3H6ykUrf9lAjgRSnkA/nOuZzA87fwl8QOM2sBEPhe4EE2kVrrcmYj/v798xneJ4uxH6/lpvELyN+j235KeAt5KTjntgObzezswKRLgRXALODOwLQ7gXdDnU3kZDVIiON3N/TguZt7smp7CVc9N5d/Ltt24heK1FPmxU1GzKwnMBFIANYBd+EvqDeB1sBG4Ebn3HHHM/b5fC43Nze4YUVqaeOufTw07QuW5Bdxa//W/OzqzjRIiPM6lsi3mFmec85X47xwvvOUSkHqm4OVVfzhX6t4cc46WqYn8/SwrlxyTo3nTIh45niloCuaRepQQlwMT17VmTe/dx7JCbGMmpLLfa/maThuCRsqBZEg6Ncug9kPDeLH3zmb/6wsYPAfP+Wleet1Ex+p91QKIkGSEBfDAxd35F+PXEDvNo156r0VDBs7T8NkSL2mUhAJsjZNGjL1rr785ZZebC8uY+jYz/jlrOWUlFV4HU3kW1QKIiFgZny3x5n8+4cXclv/Nkydv4HBf/yU2cu2Ec4ne0jkUSmIhFBacjxPD+vG2/cNoEnDRO5/bTGjpixi825d9Cb1g0pBxAO9Wjdm1oMD+dnVnclZv5vL/vQpL3y6lgoNlSEeUymIeCQuNoa7B7Xnwx9eyKBOmfzmnyu55s+fkbfxuNdsigSVSkHEYy3Tk5kwwsf4O/pQUlbB9c/P58m3l7J3/0Gvo0kUUimI1BOXdz2DD394IfcMasebuflc+odP+dsX+ToQLSGlUhCpRxomxvHTq7sw68GBZGU04JHpS7h9Ug7rCku9jiZRQqUgUg91PTONt+8bwNNDu7J0cxFXPDeXZ/+9mvLKQ15HkwinUhCpp2JjjDvOa8tHj17Id7qewbP//oYrn53L52t1pzcJHpWCSD3XrFESf7mlF1NH9aOyynHrhBx+OP1L3R9agkKlIBImLjwrk389cgEPXtyR95Zu5ZI/fMobCzdRpUH2pA6pFETCSFJ8LD/6ztnMfmgQZ5+RyhNvL+Om8fNZvaPE62gSIVQKImGoU/NUpo/J5nc3nMuaglKuem4uv31/JQcO6kC0nB6VgkiYMjNu9LXio0cvYlivljz/yVouf/ZTPlyxQ9c2yClTKYiEuYyGCTwzvAdvjMkmITaGe17OZdjYeXy8skDlICdNpSASIbLbN+H9hy/gd9efy659B7lryiKGjfucj1epHKT2LJz/svh8Ppebm+t1DJF6p+JQFTPz8vnLf9awZe8BerVO5+HBZ3FBp6aYmdfxxGNmluec89U4T6UgErkOVlbxVl4+Yz/2l0OfNo15eHAnzu+ocohmKgWRKFdeeYgZuf5y2FZUhq9NYx657CwGdGiicohCKgURAfzl8OaizYz9eC3bi8vo1zaDhy/rxIAOTb2OJiGkUhCR/1JWcYjpizYz7pM17Cgup3+7DB657Cyy2zfxOpqEgEpBRGpUVnGINxZuYtwnaykoKee89k145LKz6Ncuw+toEkTHKwXPTkk1s1gz+8LM/h543s7McsxsjZlNN7MEr7KJRIuk+FhGDmzHnMcu5hfXdGFNYSk3vjif2yYuYNEG3RY0Gnl5ncIPgK+rPf8t8CfnXEdgDzDak1QiUSgpPpZR57djzo8v5mdXd2bV9lKGvzCf2yfm6J7RUcaTUjCzLOBqYGLguQGXAG8FFpkKDPMim0g0S06I5e5B7Zn72MX89KrOfL2tmOufn88dk3JYvGmP1/EkBLzaUngWeAyoCjxvAux1zlUGnucDLWt6oZmNMbNcM8stLCwMelCRaJScEMs9F7Rn7uMX8+SV57B8azHXjfucOycv5MvNe72OJ0EU8lIws2uAAudc3qm83jk33jnnc875MjMz6zidiFTXICGO713YgbmPXczjV5zD0vy9DBs7j7teWsgSlUNE8mJLYSAwxMw2AG/g3230HJBuZnGBZbKALR5kE5EaNEyM476LOvDZ45fw2BVn88XmvQwdO49RUxaxLL/I63hShzw9JdXMLgJ+5Jy7xsxmADOdc2+Y2QvAUufcuOO9XqekinijtLySqZ9vYPycdRQdqGBw52Y8PPgsurVM8zqa1EK9vU7hqFJoj3/LIQP4ArjdOXfcm9CqFES8VVJWwZR5G5gwdx3FZZX0ap3O8D6tuKZHCxolxXsdT46h3pbC6VIpiNQPxWUVTF+4mRl5m1m9o5Sk+Biu6HoGw32tOK99E2JiNL5SfaJSEJGQcM6xNL+IN3M3M2vJVkrKKmmZnsz1fbIY3ieLVhkNvI4oqBRExANlFYf4YPl23srL57M1O3EOsttnMLxPK67sfgYNEuJO/CYSFCoFEfHU1r0HeHtxPjPy8tm4az8piXFc3b0Fw31Z9GnTWMN3h5hKQUTqBeccizbsYUbuZv6xbBv7Dx6ifdOGXN8ni+t7Z3FGWpLXEaOCSkFE6p195ZXMXraNGXn5LFy/mxiDQZ0yGe7LYnDn5iTFx3odMWKpFESkXtuwcx8zF+czMy+frUVlpCXHM7TnmQzv04puLRtp91IdUymISFg4VOX4fO1OZuTm8/7y7RysrOKcM1K5oU8W1/ZqSZOURK8jRgSVgoiEnaIDFby3ZCsz8vJZsnkvcTHGJec0Y7ivFRednUl8rJcj/4c3lYKIhLXVO0p4Ky+ftxdvYWdpOU1TEri2V0uG+1pxVvNUr+OFHZWCiESEikNVfLqqkBl5m/no6wIqqxw9stK4snsLsts3oduZjYjTFsQJqRREJOLsKi3nnS+38lZePl9vKwagYUIsfdpm0L9dBtntM+jeMp2EOJXE0VQKIhLRCkrKWLh+NznrdpOzfherd5QCkBwfS582jenfLoP+7ZvQo1UaiXE61VWlICJRZVdpub8k1u9mwbpdrNxeAkBiXAy9WqfTv10Tsts3oVfr9Ki8HkKlICJRbe/+gyxcv5sFgS2JFduKcQ4SYmPo2Sqd/u0z6N+uCX3aNCY5IfJLQqUgIlJN0YEKcjf4tyRy1u1i2ZYiqhzExxrnZqUf2d3ka9OYhomRN3CfSkFE5DhKyirI3bjnyDGJZflFVFY5YmOMbi3TyG6fQXa7JvjaNiY1Am4epFIQETkJ+8orydu4h5z1u8hZt5sl+XupOOSIMeh6Ztp/bUmkN4gPu2E4VAoiIqfhwMFDfLFpDwvW7WLB+t18uXkvByurAGiQEMsZjZJo3iiJFmlJNE9LOvL8jDT/tKYpicTWo7vPHa8UIm9nmYhIHUtOiGVAx6YM6NgU8N9A6MvNe1mWX8T24jK2F5WxvbiMnPW7KSgpo+LQf/+yHRtjZKYk0jwtiRaBsvCXRmKgTJI5o1FSvTjIrVIQETlJSfGxZLf3n9Z6tKoqx659B9lRrSwOf99RXMbawlLmrd1JSVnlt17bKCnuSGG0OLzFUW3Lo0VaEhkNE4K6u0qlICJSh2JijMzURDJTE+nWMu2Yy+0rr/QXRaAwthWVHSmSHcVlrNpews7ScqqO2sOfEBtDs0aJjBzQlrsHta/z/CoFEREPNEyMo0NmCh0yU465TOWhKgpLy48UxbbDWxxFZWSmBmcYcZWCiEg9FRcbQ4u0ZFqkJYfsMzVSlIiIHKFSEBGRI1QKIiJyRMhLwcxamdnHZrbCzJab2Q8C0zPM7EMz+ybwvXGos4mIRDsvthQqgUedc12AbOABM+sCPAF85JzrBHwUeC4iIiEU8lJwzm1zzi0OPC4BvgZaAkOBqYHFpgLDQp1NRCTaeXpMwczaAr2AHKC5c25bYNZ2oPkxXjPGzHLNLLewsDA0QUVEooRnpWBmKcBM4GHnXHH1ec4/Sl+NI/U558Y753zOOV9mZmYIkoqIRA9PLl4zs3j8hfCac+7twOQdZtbCObfNzFoABSd6n7y8vJ1mtvEUYzQFdp7ia8OV1jk6aJ2jw+msc5tjzQh5KZh/JKdJwNfOuT9WmzULuBP4TeD7uyd6L+fcKW8qmFnusYaOjVRa5+igdY4OwVpnL7YUBgJ3AMvM7MvAtJ/gL4M3zWw0sBG40YNsIiJRLeSl4Jz7DDjWuK+XhjKLiIj8t2i+onm81wE8oHWODlrn6BCUdQ7r23GKiEjdiuYtBREROYpKQUREjoj4UjCzK8xslZmtMbNvjadkZolmNj0wPydwlXVYq8U6/zAwIOFSM/vIzI55znK4ONE6V1vuejNzZhb2py/WZp3N7MZqg0++HuqMda0Wf7dbBwbc/CLw9/sqL3LWFTObbGYFZvbVMeabmf058Oex1Mx6n/aHOuci9guIBdYC7YEEYAnQ5ahl7gdeCDy+GZjude4QrPPFQIPA4/uiYZ0Dy6UCc4AFgM/r3CH4OXcCvgAaB5438zp3CNZ5PHBf4HEXYIPXuU9znS8AegNfHWP+VcA/8Z/RmQ3knO5nRvqWQj9gjXNunXPuIPAG/oH3qqs+EN9bwKWBC+zC1QnX2Tn3sXNuf+DpAiArxBnrWm1+zgBPA78FykIZLkhqs873AGOdc3sAnHMnHCWgnqvNOjugUeBxGrA1hPnqnHNuDrD7OIsMBV52fguA9MCIEKcs0kuhJbC52vP8wLQal3HOVQJFQJOQpAuO2qxzdaPx/6YRzk64zoHN6lbOuX+EMlgQ1ebnfBZwlpnNM7MFZnZFyNIFR23W+ZfA7WaWD8wGvh+aaJ452X/vJ+TJ2EdSP5jZ7YAPuNDrLMFkZjHAH4GRHkcJtTj8u5Auwr81OMfMujvn9noZKshuAaY45/5gZucBr5hZN+dcldfBwkWkbylsAVpVe54VmFbjMmYWh3+Tc1dI0gVHbdYZMxsM/BQY4pwrD1G2YDnROqcC3YBPzGwD/n2vs8L8YHNtfs75wCznXIVzbj2wGn9JhKvarPNo4E0A59x8IAn/wHGRqlb/3k9GpJfCIqCTmbUzswT8B5JnHbXM4YH4AG4A/uMCR3DC1AnX2cx6AS/iL4Rw388MJ1hn51yRc66pc66tc64t/uMoQ5xzud7ErRO1+bv9Dv6tBMysKf7dSetCmLGu1WadNxEYLsfMOuMvhUi+8cosYETgLKRsoMj9731pTklE7z5yzlWa2YPAB/jPXJjsnFtuZr8Ccp1zs/CP2PqKma3Bf0DnZu8Sn75arvPvgRRgRuCY+ibn3BDPQp+mWq5zRKnlOn8AXG5mK4BDwI+dc2G7FVzLdX4UmGBmj+A/6DwynH/JM7Np+Iu9aeA4yf8B4gGccy/gP25yFbAG2A/cddqfGcZ/XiIiUsciffeRiIicBJWCiIgcoVIQEZEjVAoiInKESkFERI5QKYicgJmVnubrHzazBtWezzaz9NMOJhIEOiVV5ATMrNQ5l3Kc+Yb/31KNQykErqL2Oed2BimiSJ3RloLIKTCztoFx/V8GvgJamdnzZpYbuHfBU4HlHgLOBD42s48D0zYErjA+fG+LrwJfD3u0OiJHaEtB5ARq2lII3IxpHTAgMGQxZpbhnNttZrHAR8BDzrmlR28pHH4OtAGm4B+LyYAc4Hbn3BehWC+RmmhLQeTUbTxcCAE3mtli/De26Yr/Ji/Hcz7wN+fcPudcKfA2MCg4UUVqJ6LHPhIJsn2HH5hZO+BHQF/n3B4zm4J/MDaRsKItBZG60Qh/SRSZWXPgymrzSvAP3320ucAwM2tgZg2BawPTRDyjLQWROuCcW2JmXwAr8d8Ja1612eOB981sq3Pu4mqvWRzYolgYmDRRxxPEazrQLCIiR2j3kYiIHKFSEBGRI1QKIiJyhEpBRESOUCmIiMgRKgURETlCpSAiIkf8fy2tq4Rcf8McAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = [i * 0.1 for i in range(11)]\n",
    "y = losses\n",
    "plt.plot(x, y)\n",
    "plt.xlabel('l ratio')\n",
    "plt.ylabel('loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "در این دیتاست، lasso نتیجه بهتری داده‌است.\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEJCAYAAACDscAcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAuvElEQVR4nO3deXxV1bn/8c83E2QiBIiMMqoMAjIcmbSOtM5FrVVUQFHEoRaqHWy9bfXq7a39qa1WRUWliqLWKk5XHFonlDIYIBBGZZBAQAkzJEACPL8/zkZjGiCBnOwMz/v14mXOPnuv8yxBvu619llLZoZzzjlXUXFhF+Ccc6528eBwzjlXKR4czjnnKsWDwznnXKV4cDjnnKsUDw7nnHOVEtPgkHSLpIWSFkh6QVJDSZMkLQ2OTZCUGJybKelVSfMlzZLU/QBtStIfJH0uabGkMbHsg3POue+KWXBIag2MASJm1h2IB4YCk4AuQA8gGRgVXHI7kGNmPYERwIMHaPpq4Gigi5l1BV6MVR+cc879p4RqaD9ZUgmQAqw1s/f2vylpFtAmeNkNuAfAzJZIai+puZl9XabNG4ErzGxfcO76QxXRrFkza9++/RF3xjnn6pPZs2dvMLOsssdjFhxmli/pPiAP2Am8VyY0EoHhwNjg0DzgYuATSf2AdkRDpWxwdAIuk3QRUACMMbMvDlZL+/btyc7OroJeOedc/SFpVXnHYzlUlQkMAToArYBUScNKnTIOmGpmnwSv7wEaS8oBfgrMBfaW03QDYJeZRYAngAkH+PzRkrIlZRcUFFRFl5xzzhHbyfHBwEozKzCzEmAyMAhA0h1AFnDr/pPNbJuZjTSzXkTnOLKAFeW0uyZoC+BVoGd5H25m480sYmaRrKz/uNNyzjl3mGIZHHnAAEkpkgScCSyWNAo4C7h8/zwFgKTGkpKCl6OI3o1sK6fd14DTg59PBT6PVQecc879p1jOccyU9DIwB9hDdOhpPFAIrAKmR/OEyWZ2F9AVeEaSAQuBa/e3JWkKMMrM1hId0pok6RZgB98+leWcc64aqD4sqx6JRMwnx51zrnIkzQ7mk7/DvznunHOuUjw4nHPOVYoHx0F8sORrXvpsddhlOOdcjRLrb47XWmbGpBl5fPx5AW2bpjCgY9OwS3LOuRrB7zgOQBJ/GdqLtk1TuGnSHFZvKgq7JOecqxE8OA6iUcNEnhwRoWTvPq6bmE3h7j1hl+Scc6Hz4DiEjllpPHxFHz7/eju3vpTDvn11//Fl55w7GA+OCjj1uCxuP7cr7y78mgffP+h6is45V+f55HgFXXtyB5Z8tZ0H3/+CLi3SOadHy7BLcs65UPgdRwVJ4g8XdadP28bc+tI8Fq0tbxkt55yr+zw4KqFBQjyPDe9LRnIi103MZsOO3WGX5Jxz1c6Do5KOSm/I+BF92bBjNzc9N4fiPfsOfZFzztUhHhyHoWebxvy/S3oy68tN3PHGQurDQpHOObefT44fpiG9WrP0q+2M+2g53VqmM3xg+7BLcs65auF3HEfgFz/ozJldjuLONxfx72Ubwi7HOeeqhQfHEYiLEw8M7UWHZqnc9Pwc8jb6siTOubrPg+MIpQfLkpjBqImfscOXJXHO1XExDQ5Jt0haKGmBpBckNZQ0SdLS4NgESYnBuZmSXpU0X9IsSd0P0fZfJe2IZf0V1b5ZKuOu7MPygkJu+bsvS+Kcq9tiFhySWgNjgIiZdQfigaHAJKAL0ANI5ts9w28HcsysJzACePAgbUeAzFjVfjhOOqYZvzuvK/9c9DV/+dfnYZfjnHMxE+uhqgQgWVICkAKsNbMpFgBmAW2Cc7sBHwCY2RKgvaTmZRuUFA/cC/wqxrVX2lWD2nNZ5Gge+mAZb85bG3Y5zjkXEzELDjPLB+4D8oB1wFYze2//+8EQ1XDgneDQPODi4L1+QDu+DZXSbgbeMLN1sar9cEnirguPJ9Iuk1++PI8F+VvDLsk556pcLIeqMoEhQAegFZAqaVipU8YBU83sk+D1PUBjSTnAT4G5wN4ybbYCfgw8VIHPHy0pW1J2QUHBkXanwhokxPPosL40SUli9MRsCrb7siTOubollkNVg4GVZlZgZiXAZGAQgKQ7gCzg1v0nm9k2MxtpZr2IznFkASvKtNkbOAZYJulLIEXSsvI+3MzGm1nEzCJZWVlV27NDyEpvwPgRETYVFXPDc7PZvWfvoS9yzrlaIpbBkQcMkJQiScCZwGJJo4CzgMvN7JuFniQ1lpQUvBxF9G7kO0vQmtlbZtbCzNqbWXugyMyOiWEfDlv31hnc/+NezF61md+9tsCXJXHO1RmxnOOYCbwMzAFyg88aDzwGNAemS8qR9Pvgkq7AAklLgXOAsfvbkjQlGKaqVc7r2ZKfnnEML2Wv4el/fxl2Oc45VyVUH/5POBKJWHZ2diifvW+fcf1zs/lgyXqeGdmPk49tFkodzjlXWZJmm1mk7HH/5niMxcWJv1zWi2Oy0vjJ83P4ckNh2CU559wR8eCoBmkNEnjyqghxglETs9m+qyTskpxz7rB5cFSTo5uk8MiVfVi5oZCxL+aw15clcc7VUh4c1WhQp2bceUE3PliynvveWxp2Oc45d1h8I6dqNmxAOxZ/tZ1HP1pOlxbpDOnVOuySnHOuUvyOo5pJ4s4Ljqdfhyb86uX5zF+zJeySnHOuUjw4QpCUEMejV/ahWVoDRk+czfptu8IuyTnnKsyDIyRN0xrwxIgIW3eWcP1zs9lV4suSOOdqBw+OEHVr1Yg/X3oCc/O28FtflsQ5V0t4cITsnB4t+dngY3l59hqe+nRl2OU459wheXDUAGPOOJZzurfgf6cs5uPPq28JeOecOxweHDVAXJy4/9IT6NyiETc/P4cVBTViK3XnnCuXB0cNkZKUwBMj+pIYH8eoidls82VJnHM1lAdHDdImM4VHr+xD3sYixrww15clcc7VSB4cNUz/jk25a0h3PlpawO9e9yetnHM1jy85UgNd0b8t+VuKeOTD5TRLTeLWH3QOuyTnnPuGB0cN9YsfdGbjjmL++sEymqQmcfVJHcIuyTnngBgPVUm6RdJCSQskvSCpoaRJkpYGxyZISgzOzZT0qqT5kmZJ6n6ANsu9vq6RxP9c2J2zjm/OnW8u4vWc/LBLcs45IIbBIak1MAaImFl3IB4YCkwCugA9gGRgVHDJ7UCOmfUERgAPHqDpA11f5yTEx/Hg0N7079CEn780z7/j4ZyrEWI9OZ4AJEtKAFKAtWY2xQLALKBNcG434AMAM1sCtJfUvGyDB7m+TmqYGM8TV0U4rnk6Nzw7m7l5m8MuyTlXz8UsOMwsH7gPyAPWAVvN7L397wdDTMOBd4JD84CLg/f6Ae04SCiUc32d1ahhIs9c04+jGjVg5NOfsWz99rBLcs7VY7EcqsoEhgAdgFZAqqRhpU4ZB0w1s0+C1/cAjSXlAD8F5gIHWzK27PVlP3+0pGxJ2QUFtX+IJyu9Ac9e05/E+DiGPzWLtVt2hl2Sc66eiuVQ1WBgpZkVmFkJMBkYBCDpDiALuHX/yWa2zcxGmlkvonMcWcCK8hou7/qyzGy8mUXMLJKVlVVFXQpX26YpPDOyHzt27WH4UzPZXFgcdknOuXoolsGRBwyQlCJJwJnAYkmjgLOAy81s3/6TJTWWlBS8HEX0bmJb2UYPdH190a1VI568KsLqzTsZ+fRnFO7eE3ZJzrl6JpZzHDOBl4E5QG7wWeOBx4DmwHRJOZJ+H1zSFVggaSlwDjB2f1uSpkhqFbw80PX1Rv+OTXn48t7MX7OFGyfNoXhPvctP51yIVB+WtIhEIpadnR12GVXupc9W86tX5vPDE1rxwGW9iItT2CU55+oQSbPNLFL2uH9zvBa79MSj2VhYzJ/eWUKT1CTuuKAb0VFB55yLHQ+OWu6GUzuyccdunvx0Jc3Skrj5jGPDLsk5V8d5cNRykrj93K5sKizmvvc+p0lqA67o3zbsspxzdZgHRx0QFyf+dElPNhcV89vXcslMSeScHi3DLss5V0f5fhx1RGJ8HOOu7EvvtpmMfTGHfy/fEHZJzrk6yoOjDklOiuepqyK0b5bC6ImzWZC/NeySnHN1kAdHHdM4JYmJ1/QnIzmRqybMYuWGwrBLcs7VMR4cdVCLjIZMvLYfBgx/aiZfb9sVdknOuTrEg6OO6pSVxtMjT2RzYTFXTZjF1p0lYZfknKsjPDjqsJ5tGjN+RIQVBYWMeuYzdpUcbLFh55yrGA+OOu6kY5rxl8t6kb1qMzc/P4c9e31dK+fckfHgqAfO69mSu4Z051+L1/PrybnUh/XJnHOx418ArCeGD2jHph3F/OVfn9M0NYnfnNs17JKcc7WUB0c9MubMY9hYuJvHp66gSWoS15/aKeySnHO1kAdHPSKJOy84nk2Fxfzx7eiKuj+OHB12Wc65WsaDo56JixN/vrQXW3eW8OvJuWSmJDG4W/Owy3LO1SIxnRyXdIukhZIWSHpBUkNJkyQtDY5NkJQYnJsp6VVJ8yXNktT9AG12kDRT0jJJfy+13ayroKSEOB4d1pfurRrxk+fn8NmXm8IuyTlXi8QsOCS1BsYAETPrDsQDQ4FJQBegB5BMdH9xgNuBHDPrCYwAHjxA038C/mJmxwCbgWtj1Ye6LK1BAhOuPpHWmclc8/RnLF73H9u7O+dcuWL9OG4CkCwpAUgB1prZFAsAs4A2wbndgA8AzGwJ0F7Sd8ZQFN3e7gyie5kDPANcGOM+1FlN0xow8Zp+pCYlcNWEWazeVBR2Sc65WiBmwWFm+cB9QB6wDthqZu/tfz8YohoOvBMcmgdcHLzXD2jHt6GyX1Ngi5ntCV6vAVrHqg/1QZvMFCZe24/de/Yx/KmZbNixO+ySnHM1XCyHqjKBIUAHoBWQKmlYqVPGAVPN7JPg9T1AY0k5wE+BucBhr5EhabSkbEnZBQUFh9tMvXBc83QmXH0iX23bxdV/m8X2Xb6ulXPuwGI5VDUYWGlmBWZWAkwGBgFIugPIAm7df7KZbTOzkWbWi+gcRxawokybG4mGy/6nwdoA+eV9uJmNN7OImUWysrKqsFt1U992mTw6rC9L1m3nmqc/8/Bwzh1QLIMjDxggKSWYmzgTWCxpFHAWcLmZfbNwkqTGpZ6QGkX0buQ7M7bBvMiHwCXBoauA12PYh3rl9M5H8eDQ3szJ28Lwp3xFXedc+WI5xzGT6CT2HCA3+KzxwGNAc2C6pBxJvw8u6QoskLQUOAcYu78tSVMktQpe3gbcKmkZ0TmPp2LVh/rovJ4tGXdlHxau3coVT8xgc2Fx2CU552oY1YcF7yKRiGVnZ4ddRq3y4dL13PDsbNo3TeW5Uf3JSm8QdknOuWomabaZRcoe99VxXblO73wUf7v6RPI2FXHZ+Ol8tdV3EXTORXlwuAMadEwzJl7bj/XbdnPp49NZs9m/5+Gc8+Bwh3Bi+yY8e20/thQVc9njM1i1sTDskpxzIfPgcIfUu20mz183gKLiPVz6+HSWrd8RdknOuRB5cLgK6d46gxdHD2TvPmPo+Oks+crXtnKuvvLgcBXWuUU6L44eSHycGDp+Bgvyt4ZdknMuBB4crlKOOSqNl64fSGpSApc/MYO5eZvDLsk5V808OFyltWuayks3DKRJahLDnpzJrJW+n4dz9YkHhzssrRsn89L1A2mR0ZCrJsxi2rINYZfknKsmHhzusDVv1JAXRw+kXdMURj79GR8uWR92Sc65auDB4Y5IVnoDXrhuAMc1T2P0s9m8u/CrsEtyzsWYB4c7YpmpSUwaNYDurTO4adIc3py3NuySnHMx5MHhqkRGciLPXtufvm0zGfviXF6evSbskpxzMeLB4apMWoMEnr7mRAZ1asYvX57H8zPzwi7JORcDHhyuSqUkJfDkVRFOOy6L21/N5elpK8MuyTlXxTw4XJVrmBjP48MjnHV8c+58cxGPfbw87JKcc1XIg8PFRFJCHA9f0YcLTmjFPW8v4cF/fUF92DTMufogpsEh6RZJCyUtkPSCpIaSJklaGhybICkxODdD0puS5gXXjDxAm5dLypU0X9I7kprFsg/u8CXGx/HAZb34UZ82/OVfn3Pvu0s9PJyrAyoUHJLGSmqkqKckzZH0g0Nc0xoYA0TMrDsQDwwFJgFdgB5AMjAquOQnwCIzOwE4DbhfUlKZNhOAB4HTzawnMB+4uWJddWGIjxP3XtKTK/q3ZdxHy7n7/xZ7eDhXy1X0juMaM9sG/ADIBIYD91TgugQgOfgLPwVYa2ZTLADMAtoE5xqQLklAGrAJ2FOmPQW/UoPzGgH+pYEaLi5O/OHC7ow8qT0Tpq3kt68tYN8+Dw/naquECp6n4J/nAs+a2cLgL+4DMrN8SfcBecBO4D0ze++bBqNDVMOBscGhh4E3iAZBOnCZme0r02aJpBuBXKAQ+ILonYqr4STx+/O70SAhnsc+Xs7uPfv40496Eh930D9GzrkaqKJ3HLMlvUc0ON6VlA7sO9gFkjKBIUAHoBXRu4RhpU4ZB0w1s0+C12cBOcG5vYCHJTUq02YicCPQOzhvPvCbA3z+aEnZkrILCgoq2E0XS5K47ezO/Gzwsbw8ew23/D2HPXsP+sfIOVcDVTQ4rgV+DZxoZkVAIlDu5HUpg4GVZlZgZiXAZGAQgKQ7gCzg1lLnjwQmB6NYy4CVROdCSusFYGbLg6Gul/a3WZaZjTeziJlFsrKyKthNF2uS+Nng4/jV2Z15Y95abn5+LsV7PDycq00qGhwDgaVmtiW4a/gtcKjt3/KAAZJSgmGtM4HFkkYRvbu4vMxQVF5wDpKaA52BFWXazAe6SdqfBN8HFlewD64Guem0Y/jd+d14Z+FX3PDcbHaV7A27JOdcBVU0OB4FiiSdAPwcWA5MPNgFZjYTeBmYQ3ROIg4YDzwGNAemS8qR9PvgkruBQZJygfeB28xsA4CknKDNtcB/A1MlzSd6B/K/FeyDq2GuPbkD/3Nhdz5Ysp7rJmazs9jDw7naQBV5NFLSHDPrE/wln29mT+0/FvsSj1wkErHs7Oywy3AH8I/s1dz2ynwi7ZvwxIgIGcmJYZfknAMkzTazSNnjFb3j2C7pN0SfgnpLUhzReQ7njtiPI0fzwNDezM3bzEXjpvHlhsKwS3LOHURFg+MyYDfR73N8RfS7F/fGrCpX7/zwhFY8e21/NhUWc+G4aUxfvjHskpxzB1Ch4AjCYhKQIel8YJeZHXSOw7nKGtCxKa//5CSapiYx/KmZ/P0zX5bduZqookuOXEr0W94/Bi4FZkq6JJaFufqpXdNUJt90EgM7NeW2V3L53ymL2evfMneuRqnoN8f/i+h3ONYDBI/D/ovoU1POVamM5ET+dvWJ3PV/ixg/dQUrCnbwwNDepDWo6B9X51wsVXSOI25/aAQ2VuJa5yotIT6Ou4Z0564hx/Ph0gIuefTf5G/ZGXZZzjkq/pf/O5LelXS1pKuBt4ApsSvLuagRA9vzt6tPJH/zToY8PI05eZvDLsm5eq+ik+O/JPrlvZ7Br/FmdlssC3Nuv1OOy2LyTYNISYpn6PgZvJ6TH3ZJztVrFfoCYG3nXwCsGzYVFnPDs7OZ9eUmxpx5LLcMPpZDLNLsnDsCh/UFQEnbJW0r59d2SdtiV65z/6lJahLPjurHJX3b8Nf3v+DmF+b6GlfOheCgj6mYWXp1FeJcRTRIiOfeS3pyzFFp/OmdJazZVMQTIyIc1ahh2KU5V2/4k1Gu1pHEDad24rFhffn86x0MeWQaC9cearFm51xV8eBwtdZZx7fg5RsHAvDjx6bz3sKvQq7IufrBg8PVase3yuD1n5zEsc3Tuf652Tz28XLqwwMfzoXJg8PVekc1asjfRw/g3B4tueftJfzy5fm+q6BzMeRrOLg6oWFiPA9f3ptjstJ48P0vyNtYxGPD+9IkNSns0pyrc/yOw9UZkrjl+8fx18t7k7NmCxc+Mo0vvt4edlnO1TkxDQ5Jt0haKGmBpBckNZQ0SdLS4NgESYnBuRmS3pQ0L7hm5AHaTJI0XtLnkpZI+lEs++Bqnx+e0IoXRw+gqHgvF4/7Nx9/XhB2Sc7VKTELDkmtgTFAxMy6A/HAUKL7enQBegDJwKjgkp8Ai8zsBOA04H5J5Y0z/Bew3syOA7oBH8eqD6726tM2k9dvPonWmclc8/RnPPPvL8Muybk6I9ZDVQlAsqQEIAVYa2ZTLEB0j482wbkGpCu6hkQasAnYU06b1wB/BDCzfWa2IcZ9cLVU68bJvHzjIE7vnMUdbyzkd68tYM9enzR37kjFLDjMLB+4D8gD1gFbzey9/e8HQ1TDgXeCQw8DXYG1QC4w1sy+81+5pMbBj3dLmiPpH5Kax6oPrvZLa5DA48MjjD6lI8/OWMXIpz9j686SsMtyrlaL5VBVJjAE6AC0AlIlDSt1yjhgqpl9Erw+C8gJzu0FPCypUZlmE4jeofzbzPoA04mGU3mfP1pStqTsggIf467P4uPE7ed25U8/6sH05Ru5eNw0Vm0sDLss52qtWA5VDQZWmlmBmZUAk4FBAJLuALKAW0udPxKYHIxiLQNWEp0LKW0jUBS0BfAPoE95H25m480sYmaRrKysquqTq8UuO7Etz43qz8bCYi58ZBozV2wMuyTnaqVYBkceMEBSSjBvcSawWNIooncXl5cZisoLziEYfuoMrCjdYDAv8ibRyXOC8xfFsA+ujhnQsSmv3XQSTVKTGPbUTF7KXh12Sc7VOrGc45hJdE/yOUTnLOKIbgb1GNAcmC4pR9Lvg0vuBgZJygXeB27bP/EtKadU07cBd0qaT3SO5Oex6oOrm9o3S2XyTScxoGNTfvXyfP44ZTF79/kyJc5VlG/k5Oqtkr37uOvNRTw7YxXfO7YZ9//4BF+e3blSDmsjJ+fqssT4OO6+sDt/vLgHn325ibMemMo/F30ddlnO1XgeHK7eu7xfW/7vp9+jVeNkrpuYzX+9msvOYt9Z0LkD8eBwDjjmqDQm3zSI60/pyKSZeZz/0CcsyPfNoZwrjweHc4EGCfH85tyuTBrVnx2793DRuGk8MXUF+3zi3Lnv8OBwroyTjmnGO2NP4fTOR/GHKYsZMWEWX2/bFXZZztUYHhzOlSMzNYnHh/fljxf3YPaqzZz9wFTe9a1pnQM8OJw7IEnRifMxJ9M6M5nrn53NbybnUlRc3tqbztUfHhzOHUKnrDQm33gS15/akRc/y+P8hz71iXNXr3lwOFcBSQlx/Oacrky6tj9Fu/dy0bhpPP7xcp84d/WSB4dzlTDomGa8PfZ7nNmlOX98ewnDnprJV1t94tzVLx4czlVSZmoSjw7rwz0X92Bu3hbOfnAq7yzwiXNXf3hwOHcYJDG0X1veGnMyR2emcMNzs/n1K/N94tzVCx4czh2BjllpvHLjIG48rRN/z17N+X/9lNw1PnHu6jYPDueOUFJCHLed3YVJo/pTVLyXix+dxmM+ce7qMA8O56rIoE7NeOdn3+P73Zpzz9tLuPLJmazbujPsspyrch4czlWhxilJPHJFH/7fJT2Zt2YLZz/wCW/nrgu7LOeqlAeHc1VMEpdGjuatMd+jXdMUbpw0h9tenk/hbp84d3VDTIND0i2SFkpaIOkFSQ0lTZK0NDg2QVJicG6GpDclzQuuGXmItt+QtCCW9Tt3JDo0S+WVGwdx02mdeGn2as5/6FPmrd4SdlnOHbGYBYek1sAYIGJm3YF4YCgwCegC9ACSgVHBJT8BFpnZCcBpwP2Skg7Q9sXAjljV7lxVSYyP41dnd+GF6wawq2QvP3r03zzy4TLf49zVarEeqkoAkiUlACnAWjObYgFgFtAmONeAdEkC0oBNwH/c20tKA24F/ifGtTtXZQZ0bMo7Y0/hrONbcO+7S7niiRms3eIT5652illwmFk+cB+QB6wDtprZe/vfD4aohgPvBIceBroCa4FcYKyZ7Sun6buB+4Gig32+pNGSsiVlFxQUHGl3nDtiGSmJPHxFb+69pCe5+Vs5+4GpvDlvLdH/h3Ku9ojlUFUmMAToALQCUiUNK3XKOGCqmX0SvD4LyAnO7QU8LKlRmTZ7AZ3M7NVDfb6ZjTeziJlFsrKyjrA3zlUNSfw4cjRTxnyPDllp/PSFuYyYMItl67eHXZpzFRbLoarBwEozKzCzEmAyMAhA0h1AFtEhp/1GApODUaxlwEqicyGlDQQikr4EPgWOk/RRDPvgXEy0b5bKKzcM5M4LujFvdfSx3bv/bxHbdpWEXZpzhxTL4MgDBkhKCeYtzgQWSxpF9O7i8jJDUXnBOUhqDnQGVpRu0MweNbNWZtYeOBn43MxOi2EfnIuZhPg4rj6pAx/+4jR+HGnDhGkrOeO+j3jps9X+rXNXo8VyjmMm8DIwh+icRRwwHngMaA5Ml5Qj6ffBJXcDgyTlAu8Dt5nZBgBJObGq07mwNU1rwB8v7skbPzmZtk1S+NUr87lo3DTm5m0OuzTnyqX6MDEXiUQsOzs77DKcOyQz47WcfP44ZQnrt+/mkr5t+NXZnTkqvWHYpbl6SNJsM4uUPe7fHHeuBpHERb3b8MEvTuOGUzvxek4+Z9z3MU9MXUHxnvIeMnSu+nlwOFcDpTVI4NfndOG9W06lX4cm/GHKYs5+cCoff+6PlrvweXA4V4N1aJbKhKtPZMLVEfbtM66aMItRz2SzamNh2KW5esyDw7la4IwuzXn3llP49TldmL58A9//81TufXeJL5zoQuHB4Vwt0SAhnhtO7cQHvziN83u25JEPl3Pm/R/zek6+f/vcVSsPDudqmeaNGvLny3rxyo0DaZaexNgXc7js8RksXOtb1rrq4cHhXC3Vt10TXv/Jyfzx4h4sK9jBBQ99ym9fy2VzYXHYpbk6zoPDuVosPk5c3q8tH/78NEYMbM8Ls1Zz2n0f8ez0L9mz1x/fdbHhweFcHZCRksidPzyeKWO+x/GtGvG71xdy/kOfMmPFxrBLc3WQB4dzdUjnFulMGtWfR6/sw/Zdexg6fgY3Pz/H9/5wVcqDw7k6RhLn9GjJv249lZ8NPpZ/LvqaM+//mIfe/4JdJXvDLs/VAR4cztVRyUnx/Gzwcbz/81M5vUsW9//zc77/l495d+FX/viuOyIeHM7VcW0yUxh3ZV+eH9Wf5MR4rn92NiMmzGL+mi1hl+ZqKV8d17l6pGTvPp6bsYq//PNztu3aw8COTbn+1I6celwW0W1znPvWgVbH9eBwrh7avquEF2et5qlPV/LVtl10aZHO6FM6csEJrUiM94EIF+XB4cHh3H8o3rOPN+et5fGpy/n86x20zGjItSd3YGi/tqQ1SAi7PBeyUIJD0i3AKMCI7gI4EngKiAAlwCzgejMrkZQBPAe0BRKA+8zsb2XaSwH+AXQC9gJvmtmvD1WHB4dzB2dmfLS0gMenLmfGik2kN0xg2IB2jBzUnqMa+SZS9VW1B4ek1sCnQDcz2ynpJWAKsB54OzjteWCqmT0q6XYgw8xuk5QFLAVamFlxqTZTgP5m9qGkJKJbzP6vmb3NQXhwOFdx81ZvYfzUFby9YB0JcXFc1Ls1153SkWOOSgu7NFfNDhQcsb4XTQCSJZUAKcBaM3uvVFGzgDbBSwPSFZ2hSwM2Ad9ZM9rMioAPg5+LJc0pdb1zrgqccHRjHrmyD6s2FvLkJyt5KXs1f89ezeCuzbnh1I5E2jcJu0QXslgPVY0F/gDsBN4zsytLvZcIzATGmtknktKBN4AuQDpwmZm9dZC2GwNzgMFmtuJgdfgdh3OHb+OO3UycvoqJ079kc1EJfdo25vpTO/H9rs2Ji/Mnseqyat9zXFImMAToALQCUiUNK3XKOKLDVJ8Er88CcoJzewEPS2p0gLYTgBeAvx4oNCSNlpQtKbugwLfbdO5wNU1rwC3fP45pvz6D//7h8RTs2M31z85m8J8/5oVZef5t9Hools/dDQZWmlmBmZUAk4FBAJLuALKAW0udPxKYbFHLgJVE7z7KMx74wsweONCHm9l4M4uYWSQrK+vIe+NcPZeSlMBVg9rz4c9P46HLe5PaIIHfTM7l5D99yCMfLmNrUUnYJbpqEsvgyAMGSEoJ5i3OBBZLGkX07uJyM9tX5vwzASQ1BzoD/3E3Iel/gAzgZzGs3Tl3AAnxcVxwQiveuPkknh/Vn+NbNeLed5cy8J73uevNReT7gop1XqznOP4buIzoJPdcoo/mFgKrgO3BaZPN7C5JrYCngZaAgHvM7LmgnRwz6yWpDbAaWALsDq5/2MyePFgdPsfhXGwtXreNJ6au4I15azHggp4tGX1KJ7q1Kne02dUS/gVADw7nYi5/y04mfLqSF2flUVi8l+8d24wbTu3EoE5NfUmTWsiDw4PDuWqztaiE52au4m/TvmTDjt10b92I0ad04tzuLUjwJU1qDQ8ODw7nqt2ukr28Njef8Z+sYEVBIW0yk7lqYHvO69mSVo2Twy7PHYIHhweHc6HZt8/41+KvGT91BdmrNgPQu21jzuvRknN6tKS1h0iN5MHhweFcjbByQyFTctcxJXcdC9duA6DX0ftDpAVtMlNCrtDt58HhweFcjfPlhkKmLIiGyIL8aIiccHRjzuvRgnO6t+ToJh4iYfLg8OBwrkZbtbGQKblfMSV3Hbn5WwE4oU0G5/Zoybk9PETC4MHhweFcrZG3seibO5H5a6Ih0jMIkfM8RKqNB4cHh3O10upNRd/MicwLQqRH629DpG1TD5FY8eDw4HCu1lu9qYi3F6zjrdyvmLd6CwDdWzf6JkTaNU0Nt8A6xoPDg8O5OmXN5iLezv2Kt3LXkROEyPGtvg2R9s08RI6UB4cHh3N1Vnkh0q1lI87rGZ1Y7+Ahclg8ODw4nKsX8rfs5O3cdbyVu465eVsA6NqyEed2b8GJHZrQvXUGaQ1ivflp3eDB4cHhXL2zdsvObybW5wQhIkGnrDR6ts6gZ5sMerRpzPGtGtEwMT7cYmsgDw4PDufqtY07djM/fyvzV28lN38L89ZspWB7dHeG+DhxXPP0aJgcnUHP1o3p3CKdpIT6vSCjB4cHh3OuFDPj6227mbdmC7lrtkZDZc0WtgQ7GSbFx9G1ZTo92mTQs01jerbJ4JistHq1uq8HhweHc+4QzIw1m3d+Eybz1mxhQf42duzeA0ByYjzHt2r0TZD0aJNBh6apxMXVzb1GPDg8OJxzh2HfPmPlxkLmr9nC/DVbmb9mKwvXbmVXSXTn6/QGCXQvNcTVs00GbTKT68TGVQcKjpg+WiDpFqLbxRqQC4wEngIiQAkwC7jezEokZQDPAW2Duu4zs7+V02ZfolvMJgNTgLFWH9LPOReKuDjRKSuNTllpXNS7DQB79u7ji/U7vrkryc3fyoRPV1KyN/pXUWZKIj3aNOaENhn0aJ1Bx6w0mqQmkZGcSHwduDuJ2R2HpNbAp0A3M9sp6SWif9GvB94OTnsemGpmj0q6Hcgws9skZQFLgRZmVlym3VnAGGBm0N5fzextDsLvOJxzsbZ7z16WfrU9uCuJ3p18sX4He/d9+3esBBnJiTRJSSIzNYnMlCSapCaSmZoUPRYcb5KaGLyXRKOGiaENhYVyxxG0nyypBEgB1prZe6WKmgW0CV4akK7o/V0asAnYU7oxSS2BRmY2I3g9EbiQb4PIOedC0SAhPpj7aAy0A2Bn8V4WrdvKms072VxYzKaikuCfxWwpKiZ/y04W5G9lU2ExxXv3ldtunKBxShKZKYk0+SZskr4Jm8b7j+8Pn9QkGjVMiOlQWcyCw8zyJd0H5AE7gffKhEYiMBwYGxx6GHgDWAukA5eZWdl/k62BNaVerwmOOedcjZOcFE/fdk3o2+7g55kZRcV72VRYzOaiYjbvD5jgdel/5m0qImf1FjYXFX8zNFZWQpxonBK9axk/IlLl35yPWXBIygSGAB2ALcA/JA0zs+eCU8YRHab6JHh9FpADnAF0Av4p6RMz23aYnz8aGA3Qtm3bw+2Gc87FnCRSGySQ2iChwkvGmxk7du9hc2EJm4qCwPlO2ETDJ7VB1X+xMZZDVYOBlWZWACBpMjAIeE7SHUAWcH2p80cC9wQT3cskrQS6EJ1A3y+fb4e2CH7OL+/DzWw8MB6icxxV0iPnnKshJJHeMJH0honVvrR8LL/JkgcMkJQSzFucCSyWNIro3cXlZYai8oJzkNQc6AysKN2gma0DtkkaELQ5Ang9hn1wzjlXRiznOGZKehmYQ3SSey7RO4BCYBUwPZi8mWxmdwF3A09LygUE3GZmGwAk5ZhZr6Dpm/j2cdy38Ylx55yrVv4FQOecc+U60OO49WfRFeecc1XCg8M551yleHA455yrFA8O55xzleLB4ZxzrlLqxVNVkgqIPgJ8OJoBG6qwnNrA+1w/eJ/rviPtbzszyyp7sF4Ex5GQlF3e42h1mfe5fvA+132x6q8PVTnnnKsUDw7nnHOV4sFxaOPDLiAE3uf6wftc98Wkvz7H4ZxzrlL8jsM551yleHAEJJ0taamkZZJ+Xc77DST9PXh/pqT2IZRZpSrQ51slLZI0X9L7kg6xj1nNd6g+lzrvR5JMUq1+Aqci/ZV0afD7vFDS89VdY1WrwJ/rtpI+lDQ3+LN9bhh1ViVJEyStl7TgAO9L0l+DfyfzJfU5og80s3r/C4gHlgMdgSRgHtCtzDk3AY8FPw8F/h523dXQ59OBlODnG+tDn4Pz0oGpwAwgEnbdMf49PpbolgeZweujwq67Gvo8Hrgx+Lkb8GXYdVdBv08B+gALDvD+uUS3oBAwAJh5JJ/ndxxR/YBlZrbCzIqBF4lue1vaEOCZ4OeXgTMVy93gY++QfTazD82sKHg5g+/uvlgbVeT3GaJ7w/wJ2FWdxcVARfp7HfCImW0GMLP11VxjVatInw1oFPycAaytxvpiwsymApsOcsoQYKJFzQAaS2p5uJ/nwRHVGlhd6vWa4Fi555jZHmAr0LRaqouNivS5tGup/ZtmHbLPwS380Wb2VnUWFiMV+T0+DjhO0jRJMySdXW3VxUZF+nwnMEzSGmAK8NPqKS1Ulf3v/aBiuee4qyMkDQMiwKlh1xJLkuKAPwNXh1xKdUogOlx1GtE7yqmSepjZljCLirHLgafN7H5JA4FnJXW3725l7Q7C7zii8oGjS71uExwr9xxJCURvcTdWS3WxUZE+I2kw8F/AD81sdzXVFiuH6nM60B34SNKXRMeC36jFE+QV+T1eA7xhZiVmthL4nGiQ1FYV6fO1wEsAZjYdaEh0Tae6rEL/vVeUB0fUZ8CxkjpISiI6+f1GmXPeAK4Kfr4E+MCCWada6pB9ltQbeJxoaNT2sW84RJ/NbKuZNTOz9mbWnui8zg/NrLbuO1yRP9evEb3bQFIzokNXK6qxxqpWkT7nAWcCSOpKNDgKqrXK6vcGMCJ4umoAsNXM1h1uYz5URXTOQtLNwLtEn8qYYGYLJd0FZJvZG8BTRG9plxGdhBoaXsVHroJ9vhdIA/4RPAeQZ2Y/DK3oI1TBPtcZFezvu8APJC0C9gK/NLNaeyddwT7/HHhC0i1EJ8qvruX/E4ikF4j+D0CzYO7mDiARwMweIzqXcy6wDCgCRh7R59Xyf1/OOeeqmQ9VOeecqxQPDuecc5XiweGcc65SPDicc85VigeHc865SvHgcK6KSNpxhNf/TFJKqddTJDU+4sKcq2L+OK5zVUTSDjNLO8j7IvrfXLlLWwTfVo+Y2YYYlehclfA7DudiSFL7YG+IicAC4GhJj0rKDva/+O/gvDFAK+BDSR8Gx74Mvs29f2+UBcGvn4XUHecAv+NwrsqUd8cRbPi1AhgULGeNpCZmtklSPPA+MMbM5pe949j/GmgHPE107SwBM4FhZja3OvrlXFl+x+Fc7K3aHxqBSyXNIbqB0vFENxM6mJOBV82s0Mx2AJOB78WmVOcOzdeqci72Cvf/IKkD8AvgRDPbLOlpoovsOVdr+B2Hc9WrEdEg2SqpOXBOqfe2E13avaxPgAslpUhKBS4KjjkXCr/jcK4amdk8SXOBJUR3ZJtW6u3xwDuS1prZ6aWumRPcmcwKDj3p8xsuTD457pxzrlJ8qMo551yleHA455yrFA8O55xzleLB4ZxzrlI8OJxzzlWKB4dzzrlK8eBwzjlXKR4czjnnKuX/A4dnHC6TG22qAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "X, y = make_regression(n_samples=5000, n_features=3, noise=3)\n",
    "y = y[:, np.newaxis]\n",
    "\n",
    "for i in range (5000):\n",
    "    if i % 4 == 0:\n",
    "        X[i] = X[i] * 20\n",
    "        y[i] = y[i] * 20\n",
    "losses = []\n",
    "for i in range(11):\n",
    "    parameters = {\n",
    "        \"l_ratio\" : i * 0.1,\n",
    "        \"l\" : 0.1,\n",
    "        \"lr\" : 0.0001,\n",
    "        \"epoch\" : 10000\n",
    "    }\n",
    "    model = ElasticNet(**parameters)\n",
    "    model.fit(X, y) \n",
    "    y_pred = model.predict(X)\n",
    "    losses.append(mse_loss(y_pred, y))\n",
    "x = [i * 0.1 for i in range(11)]\n",
    "y = losses\n",
    "plt.plot(x, y)\n",
    "plt.xlabel('l ratio')\n",
    "plt.ylabel('loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div dir=rtl id=\"dataset\">\n",
    "  <font face=\"XB Zar\" size=4>\n",
    "همانطور که مشاهده می‌کنید با توجه به نیازی که داریم و نحوه توزیع دیتاست، استفاده از هر یک از این مدل ها می‌تواند مقدار خطا را کمتر می‌کند.\n",
    "    </font>"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
